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PART ONE: 



DESCRIPTION OF THE PROJECT 



la THE PROJECT IN BRIEF 

In 1989, a new science curriculum was established for Ontario secondary schools. A 
significant component of the new science program deals with the social context of science 
and technology. This area of scholarship is commonly referred to as "STS", which s tan ds 
for "science-technology-society". It was developed significantly at the post-secondary level 
during the 1970s, and has been influencing science programs at the secondary and 
elementary levels since the early 1980s. 

This new element has been introduced into all Ontario science courses, at all grades and 
all levels. For each science topic, the teacher must now include real-world applications of 
science concepts, and consider societal implications of the related science and technology. 
At the OAC level, a new course, Science in Society, has been added. 

This research project was commissioned by the Ministry of Education to develop ah 
instrument that assesses student attitudes toward science and technology (S&T), and 
their understanding of certain social issues surrounding S&T. 

A number of objectives guided the present study: 

• to find a way to obtain a baseline measure of Ontario students’ 
understanding of STS issues (knowledge and attitudes) 

• to provide Boards, teachers, Faculties of Education and educational 
researchers with a tool for monitoring students’ understanding of STS issues 

• to build on existing research in STS evaluation 

The new science curriculum is now being implemented province-wide. It is hoped that the 
results of this study will provide educators with an effective approach to monitor their 
progress over the coming years. 

2. BACKGROUND 

2.1. The New Thrust in Science Education: STS 

A number of realities in Canada 1 today have focused the attention of educational and 
other policy makers on science and technology. 

• Science and technology are seen to be key to future economic growth. 

• Canada has one of the poorest records in research and development among 
industrial nations. 

• Scientific literacy among the adult population is low. 2 

• : Despite a general interest in the results of scientific research, people are not 
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getting enough information about S&T from the media. 3 

• Fewer students entering post-secondary school are choosing science over 
other subjects. 4 

• Science enrollments in secondary school are dropping. 6 

• Significantly fewer women than men take science courses in secondary school. 

• Canadian student performance in international math and science tests is 
lower than that of other countries.® 

These facts have led to widespread efforts to revamp science education, improve public 
attitudes toward science and technology 7 , and provide better advice to policy makers. 8 

The most significant development in the field of science education has been the 
integration of science-technology-society, or STS, in the teaching of science. The STS 
movement in education began in the 1970s at the post-secondary level in many industrial 
countries, notably the U.K., U.S. and Canada. It reached grade schools about a decade 
later. 

Canada was among the first nations to recognize the need for a renewal in elementary 
and secondary science education that would integrate science and technology into general 
Canadian culture. The Science Council of Canada initiated a national consultation during 
the early 1980s®. At the same time, individual provinces began to review their science 
curricula, with the aim of making courses more relevant to the real world. Ontario was 
one of the first provinces to introduce a new science curriculum with a significant STS 
component. 

The STS focus is particularly appropriate for secondary school science programs, since 
most students will not go on to specialized careers in science or technology. Yet the 
influence of science and technology on everyday life is enormous, and growing. Young 
people must increasingly make personal and professional choices about issues in which 
science and technology, and scientists and engineers, play a part. Therefore, an essential 
ingredient in their education must be science, its technological applications and societal 
implications. 

In response to this need, the Ontario Ministry of Education initiated a mqjor renewal of 
the secondary science program. 

23. A New Secondary Science Curriculum for Ontario 

In 1987, the Ministry of Education introduced a new science program for Grades 7 to 12 
and the Ontario Academic Courses (OACs). Twenty-eight new courses were developed to 
replace science guidelines produced in the 1960s and 1970s. The new secondary science 
program is described in a fifteen-part set of guidelines that was published throughout the 
period 1987-89. By September 1989, the new program was fully established, and the 
implementation phase was in full swing. 

The most significant change in the new science program has been the introduction of a 
strong and unifying STS focus. Now Boards of Education and individual teachers are 
required to develop and teach for all program levels - basic, general and advanced - 
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science units which include the practical, concrete and societal aspects of science and 
technology. 



The published aims of the science curriculum strongly reflect the STS philosophy. They 
focus on the relationship between science and technology themselves, and between S&T 
and the individual and society (see Table l). 10 



A primary goal of the new science curriculum is to develop students’ scientific literacy. In 
fact, the policy document makes this point quite emphatically. 11 

"Indeed, the success of a science program might be assessed on the basis of the degree to 
which scientific literacy is achieved." 

In this context, it is important to realize that scientific literacy does not consist of specific 
content knowledge. In fact, the teaching policy for the new science courses makes a clear 
distinction between the content of science and the process of science. It warns against 
overemphasizing the former over the latter, and urges teachers to foster attitudes as well 
as knowledge and skills. 12 This philosophy is consistent with an integrated STS focus, 
which presents science as a human endeavour, carried out in a social context. 



Table 1 

The Aims of the Science Curriculum 13 

1. an understanding of the processes of science 

2. skills that are essential for participation in scientific work and technology 

3. facility in problem solving through science 

4. the basic knowledge needed to function in and contribute to a scientific and 
technological world 

5. respect for the environment and a commitment to the wise use of resources 

6. an understanding of the nature of science as a human endeavour 

7. an appreciation of technology as the application of scientific knowledge and 
principles 14 

8. an ability to locate and retrieve scientific information 

9. an awareness of the career possibilities in the field of science and technology 

10. an awareness of how the knowledge of science enhances personal life 
management 

11. a sensitivity about science and its influence on societal issues and values 

The new science curriculum has been designed to develop a larger awareness of science 
and technology among the total student population. The intent of an STS focus is to 
make science courses more interesting to all students, and relevant to their lives and 
personal aspirations. Even students whose interests, abilities and career goals will not 
lead them to S&T careers, will learn more about science and technology than they did 
before. Students who want to go on to take university science and engineering will learn 
to appreciate the larger societal context in which they will study and work, in addition to 
acquiring the required cognitive . skills and knowledge. 



However, it will be essential for educators to have a way of assessing the degree to which 
students respond to the expressed aims of the new curriculum. 



O 
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2.3. Monitoring Changes in Student Understanding and Attitudes 

In order to assess how students are responding to the new curriculum, it is important to 
be able to track changes in students’ understanding of STS issues over time. A way must 
be found to obtain a basebne "snapshot" of Ontario students’ understanding at one time, 
for a range of grade levels. Comparisons between grade levels for one "snapshot" will 
reflect differences in level of education and age. Comparisons between "snapshots" taken 
in different years provides a way to assess changes over time. 

The present system of provincial curriculum reviews uses questionnaires developed for 
specific subject areas. Multiple-choice and/or essay questions are used to assess student 
comprehension of course material. 

For science, h anks of questions have been developed that measure content knowledge in 
some areas of the science curriculum. 16 However, there is now a need for question items 
that measure familiarity and comprehension of STS-related topics. This project was 
established to explore the best way to develop such a bank of questions. 

2 A. The Present Study 

Since the STS mandate is relatively new, there is no base of data against which progress 
in STS education can be measured. The objective of this project was to develop an 
instrument that can help provide such information. A very important requirement was 
the development of an instrument that can be used for Grade 10 and Grade 12 students 
with a wide range of abilities. This would allow grade comparisons within a given year, 
as well as tracking populations over time. 

2.4.1. Views on Science, Technology and Society (VOSTS) 

The starting point for the study was an instrument developed by Professor Glen 
Aikenhead of the University of Saskatchewan - "Views on Science, Technology and 
Society" (VOSTS). VOSTS is a bank of multiple-choice items designed over a period of 
years, in collaboration with graduating high school science students from across Canada. 
The VOSTS instrument is highly regarded by the international evaluation community, 
and is one of the only existing instruments for STS at the secondary level. 16 

The research that led to VOSTS began in 1984, and the first published version of the 
instrument appeared in 1987. The VOSTS question items continue to be developed and 
expanded, and now cover a wide range of STS topics. 

In order to build on this previous research, the terms of reference for the present study 
were to adapt VOSTS to the Ontario case. A number of specific requirements had to be 
taken into consideration: 

• the final instrument must be suitable for students of grade 10 and grade 12 
the final instrument must be suitable for all levels within each grade - basic, 
general and advanced 17 

• the final instrument must address the specific STS needs of the Ontario 
science curriculum 



These requirements led ultimately to a radical simplification of the VOSTS structure, and 
to an expansion of the question format to include a mixture of testing methodologies. As 
a consequence, the resulting instrument loses some of the subtler analytical capabilities of 
VOSTS that were appropriate for older, more sophisticated students. However, the 
instrument has a wider applicability to grade levels and reading and comprehension 
abilities. In subsequent sections of this report, we will return to this issue, and highlight 
ways in which the instrument may be used effectively for various purposes. 

2.4.2. Project Tasks 

A project team was put together consisting of The Impact Group, Market Facts of Canada 
and Professor Glen Aikenhead in Saskatchewan . Dr. Jeffrey Crelinsten of The Impact 
was Project Leader. Ms. Jean de Boerr of Market Facts was Senior Consultant. Other 
members of the project team were Ms. Sheila Kumar and Mr. Ron Ko of Market Facts and 
Mr. Ron Freedman of The Impact Group. 

The project team, in collaboration with the Ministry of Education, established the 
following sequence of tasks: 

1) research the field of STS evaluation 

2) correlate VOSTS with the Ontario science curriculum 

3) adapt VOSTS to design a single instrument that can measure Grade 10 and 12 Ontario 
students’ understanding and appreciation of the STS perspective. 

The goal at the outset was to develop an instrument for grades 10 and 12 students at all 
levels - basic, general and advanced - for both official languages, for urban and rural 
populations, and for public and separate schools. The development process proceeded 
through various phases, which will be described in the Sections 3 and 4. The final 
instrument was tested with students across the province. 

The resulting instrument will be useful for teacher inservice, teacher training and 
research, as well as for evaluation and monitoring. Another very interesting project 
would be to use the instrument for a province-wide survey. This would generate the first 
baseline of information on Ontario students’ knowledge and attitudes in the STS area. 
Subsequent province-wide surveys could be taken with the same instrument, and 
comparisons could help assess progress in STS teaching in Ontario over time. 

2.4.3. The Choice of the Target Group - Grades 10 and 12 

The VOSTS questionnaire was designed for graduating high school students, and tested 
nationally. Even .though students were involved in the design of the questions, the 
reading level required to understand a typical VOSTS item is rather high. Therefore, the 
Ministry decided at the outset to adapt VOSTS for Grades 10 and 12 as an initial step. 
The strategy selected was to try to design a single instrument for all levels of students 
within the two grades (basic, general, advanced), and to test the instrument in schools. 

Grades 10 and 12 represent graduates of Grades 9 and 11 respectively. The Grade 9 
cohort is important because its members provide a "control" for the analysis. It would 
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have been ideal to use grade 6 graduates (grade 7 students) as a control group, since they 
represent the "entry level" into the new intermediate and senior science curriculum. 
However, it was felt that the VOSTS questionnaire would not be easily adapted to this 
age group, and at the same time be useful at senior secondary grades. Therefore it was 
decided to use grade 10 students (grade 9 graduates) as the "entry" level group, and to 
adapt the VOSTS questionnaire to this age group. 

Ideally, OAC students (grade 12 graduates) would be used as the experimental group, 
since the original VOSTS instrument was designed with these students’ input. However, 
large numbers of students do not proceed to OAC. The grade 12 student cohort 
represents the best compromise between STS exposure and experimental practicalities. 
Grade 12 students (11th grade graduates) will have had two more years of an integrated 
STS focus in science instruction than the grade 10 (grade 9 graduates) group. The grade 
9-11 range is also appropriate since these years were ones, at the time the data were 
gathered, in which the basic, general and advanced streams applied. This allowed 
meaningful comparisons among the three sub-groups, using the modified VOSTS 
instrument. 18 

3. METHODOLOGY 

3.1. A Survey of the STS Evaluation Literature 

The international STS evaluation literature was researched as a preliminary step to the 
study, and a literature review was prepared (see Part Two). 

It was found that the use of STS in science teaching at the secondary level is growing 
rapidly in most developed countries and many developing countries. However, there is an 
extreme diversity of views among educators and policy makers around the world 
regarding why STS education is important, and exactly what constitutes good STS 
education. For example, in the U.S., the primary reason for initiating STS programs was 
a fear of losing S&T dominance to the Soviet Union, and later Japan. In the U.K., social 
activism and a rebellion against specialization in the school system have been identified 
as starting the STS movement in British schools. In developing countries, STS is an 
integral component of the development process. 

When it comes to what STS education is all about, opinions differ as widely. Some 
science educators see science content as primary, and STS in a supportive motivational 
role. Others see STS as empowering students to take critical social action, by developing 
critical thinking and social skills. Often there is conflict between various views held by 
different practitioners. The literature indicated a need to involve teachers in the process 
of STS curriculum development, and to provide support materials and ihservicing. There 
is evidence that while most science teachers support STS, many feel unqualified to teach 
it properly. 

Specific evaluation techniques used include multiple-choice questions, Likert-type (bipolar) 
instruments, essay instruments and interview instruments. The most common are the 
multiple-choice and Likert-type since the processing of the data is much easier. However, 
there are limitations to these techniques. Multiple-choice questions are often designed 
with certain assumptions, and the interpretation of the results is accordingly suspect. 
Likert-type measures of student attitudes toward STS issues are problematic, because the 
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design of these measures is uni-dimensional. They measure attitudes on a single 
continuum from "strongly agree" to "strongly disagree". STS issues, however, are often 
multi-dimensional in nature. It is therefore necessary to have controls and cross-checks 
to interpret the results of such tests properly. 

In developing VOSTS, Aikenhead used Likert-type scales, essays and interviews to derive 
empirically a series of multiple-choice items dealing with STS issues. The resulting 
instrument, Views on Science, Technology and Society (VOSTS) is a multiple-choice 
instrument that measures students’ knowledge (reasoned beliefs) about STS issues. 
Aikenhead has compared results obtained using VOSTS with those obtained using only 
Likert-type, essay or interview instruments. He found that the empirically-derived 
multiple-choice items in VOSTS provided significantly less ambiguous results than did the 
Likert and essay techniques. The interviews provided the most accurate data, but took 
the most time, and hence resources. VOSTS offered the most efficient of all modes. 

A number of specific issues have been investigated by the international STS evaluation 
community. These include gender, cultural differences, views about science and 
technology, and regional differences. Good instrument design for monitoring and 
evaluation purposes must attempt to be: 

• gender neutral 

• sensitive to cultural diversity 

• inclusive of various views regarding the nature and importance of science and 
technology ' 

• locally relevant wherever possible 
3.2. Monitoring versus Evaluation 

The diversity of views about STS itself makes evaluation difficult. Interpreters of data 
from any one instrument must bear in mind the variations that exist in the STS field 
itself, as well as the teaching and learning environments in which it is taught and 
learned. 

The best strategy for a useful instrument is to use it to establish a baseline of views from 
which to monitor progressive change. In this way, even an incomplete scan of students’ 
understanding and appreciation of STS issues can be an effective, tool for monitoring 
change. 

For this reason, the project team and its advisors decided to shift the initial emphasis on 
evaluation to one on monitoring. The latter term is more appropriate when thinking 
about how to assess progress in the STS aspects of Ontario’s science curriculum. Since 
the STS field itself is diverse and subject to varied interpretations, it would be 
counter-productive to develop an instrument that purports to evaluate students’ 
knowledge in the STS field. Rather, the instrument should attempt to provide teachers 
with a tool they can use to monitor change over time. 

This point emerged strongly in discussions with teachers during the design phase. Many 
teachers felt that STS is a new aspect of the science curriculum, and that they need tools 
to help them develop and refine their teaching. This includes familiarization with 
relevant and interesting issues, and with standard STS vocabulary. 
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The vocabulary issue is particularly relevant. STS falls into the domain of the social 
sciences and humanities, where reading comprehension is an integral part of the learning 
process. Students’ in Ontario schools have a wide range of reading ability. An 
instrument that is practical for a wide range of reading ability will be more useful to 
monitor progress in STS learning over time, rather than evaluating students’ knowledge. 

The decision to develop a monitoring instrument evolved over the early stages of the 
project. This evolution is described in more detail in the following section. 

3.3. The Starting Point - VOSTS 

The starting point for this study was the instrument developed for graduating high school 
science students - Views on Science, Technology and Society (VOSTS). This instrument 19 
was adapted through a detailed consultative process described in Section 4 of this report. 
This section elaborates some of the features of VOSTS as a tool for measuring students’ 
understanding of STS issues. 

It is important, to point out that VOSTS measures knowledge rather than attitudes. The 
STS topics included in VOSTS are limited to those emphasizing student thinking rather 
than student attitude. Attention is not given to feelings about global or regional issues. 
Instead, VOSTS focuses on the reasons that the students give to justify a viewpoint. 
Consequently VOSTS consists mainly of information viewpoints; that is, cognitive beliefs. 

The beliefs addressed by VOSTS items are limited to a domain of topic which has been 
described 20 as the "social aspects of science," in contrast to "social issues" themselves. 
That is, VOSTS does not elicit student reaction to social issues such as pollution, but 
VOSTS does monitor students’ reasoned beliefs about how science and technology 
contribute to such problems, and how science and technology may help resolve them. 

VOSTS content is, in part, based on the same theoretical models which validated the 
standardized instruments developed in earlier years of STS evaluation. 21 VOSTS content 
is also based, however, on the more recent literature concerning the social and 
technological aspects of science. 22 

The topics in VOSTS are grouped within the following categories: 

• Definitions of science, technology and R&D 

• The influence of society on science/technology 

• The influence of science/technology on society 

• The influence of school science on society 

• Personal characteristics of scientists 

• The social construction of scientific knowledge 

• The social construction of technological knowledge 

• The nature of scientific knowledge. 

Each VOSTS item is assigned a five-digit code. For example, consider item 90521 (see 
Appendix C). 




- 8 - 



15 



* The first digit (9) corresponds to section nine ("the nature of scientific 
knowledge"). 

• The next two digits (05) refer to the topic number within that mqjor section 
("hypotheses, theories, and laws"). 

• The fourth digit (2) indicated the item number within that topic. For 
instance, 90521 is the second item for topic 05 in section 9. 

* Lastly, the fifth digit differentiates items which have slight but significant 
variations in their wording, such as a different example or a different key 
word. Thus, item 90522 would have a small but significant change in its 
wording from item 90521. Only a few VOSTS items exhibit this slight 
variation. 

Teaching science through STS means that STS content is taught in conjunction with the 
normal science discipline content of facts, principles, concepts and problem-solving skills. 
Thus, VOSTS by itself does not assess an STS course. VOSTS assesses some of the STS 
objectives or goals of an STS course. 

Other limitations to VOSTS should be noted: 

VOSTS is only one strategy for monitoring students’ reasoned beliefs about STS topics; 
namely, a multiple-choice instrument. Other methods include essay writing, oral 
examinations, reports, projects, and check lists. By deleting the multiple-choice options 
from a VOSTS item, a teacher can transform a VOSTS item into an essay type exam 
question, and accurately assess students’ viewpoints on an STS topic by reading their 
paragraph responses to the VOSTS statement. This assessment, of course, depends on a 
teacher having (1) discussed the STS idea in class, (2) acquainted students with writing 
paragraphs in science class, and (3) the time to read and grade the paragraphs. A 
three-point scoring system usually works well: three points for an answer which is 
complete and to the level of sophistication one expects; two points for an answer falling a 
bit short of expectations; one point for at least writing something accurate on the topic 
(demonstrating that the classroom material was being applied); and zero points for 
answers off topic or expressing misconceptions or naive conceptions on the topic. 

Unlike the typical multiple-choice question, VOSTS items have no absolutely "right" 
answers among the choices. One must examine the choices ahead of time and decide 
which choices students should pick if students have attained the objectives of a 
curriculum. In some cases, there may even be more than one acceptable choice. Also, one 
needs to identify which choices are incorrect, naive or inappropriate. Students selecting 
these are indicating that they have not attained the curriculum’s objectives. 

Current t hinking in the STS literature will identify certain perspectives as being the most 
accurate portrayals of science in a technological and social context. These perspectives 
will, one hopes, be embraced by a science curriculum. Therefore, in terms of specific 
perspectives on STS, VOSTS items do have "right" answers. However, arguments by 
informed scholars over what qualification ought to be attached to those "right" answers 
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form the basis of more sophisticated and elaborate VOSTS choices. We will return to this 
issue when discussing the final instrument developed in the present study (Volume 2). 

The VOSTS item pool is a new tool for monitoring students’ reasoned beliefs about STS 
issues and for initiating class discussions. Because the choices for each item were derived 
empirically from students’ views on STS topics (rather than from science educators’ 
philosophical positions), a teacher can feel secure in knowing that the meaning which 
teachers read into each choice is likely the same meaning which students read into each 
choice. 

A teacher can select those VOSTS items which suit the teacher’s particular instruction. 
Although the 114 VOSTS items cover the domain of STS topics pertaining to reasoned 
beliefs, the item pool does not address all possible STS topics. Consequently, teachers can 
modify or develop their own items and add them to their personally selected pool of 
VOSTS items. " 

3.4. Using VOSTS to Fulfill Project Objectives 

The very complexity and sophistication that makes VOSTS an exciting and useful 
instrument for STS evaluation at the graduating high school level, makes it unsuitable for 
an instrument to monitor students’ STS knowledge and attitudes at lower grades. 
Nonetheless, the content of the VOSTS items served to be an excellent starting point for 
developing a monitoring instrument for Ontario schools. 

The powerful feature of a VOSTS item is the wide range of options listed for the student 
to choose from. Through careful consultations and testing with students, the VOSTS 
items have been refined to include a wide, and subtle, range of possible interpretations of 
each issue. These diverse and detailed interpretations are held by students themselves, 
and expressed in their own words. 

While this detail is an asset, it was also a liability in the context of project objectives. A 
student must read - and understand - a large amount of text for each item. This limits 
the number of items that can be administered in a typical testing situation. More serious, 
however, is the fact that students with limited reading comprehension and interest are 
handicapped by the amount of reading required to answer even one question. 

The goal of this study was to produce an instrument that could be handled with the entire 
range of abilities between basic grade 10 and advanced grade 12. This requirement made 
it impossible to use the VOSTS instrument as is. In particular, the reading level had to 
be simplified, and the number of options reduced. The total number of items was also 
reduced, as described below in Section 4. 

In addition to simplifying the existing VOSTS structure, the project team included a 
variety of other elements into the instrument design. Dining the design phase (see 
section 4.1 below) a combination of multiple-choice, essay and interview techniques was 
used. VOSTS items were selected, revised, and tested with teachers and students. An 
open-ended essay question was included for each item. And finally, for comparison, a 
battery of bipolar attitudinal statements was designed independently from the VOSTS 
items, dealing primarily with attitudes toward science, technology, and STS-related 
issues.. 
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Focus groups were held to discuss teacher and student reactions to the various items. In 
the final analysis, it was decided to develop an instrument that included simplified 
VOSTS-type items plus the battery of bipolar (Likert- type) attitudinal statements. The 
reasons for this decision will be elaborated in Section 4. 

4. DESIGN AND TEST PROCEDURES 

4.1. Summary of the Design Process 

The starting point for the design was the VOSTS instrument, consisting of over 100 items 
relating to various aspects of STS. The challenge lay in the requirement to provide one 
instrument for both grades 10 and 12, and all three levels (basic, general, advanced) 
within each grade. Since the reading ability among these different groups varies 
radically, we had to design as simple a questionnaire as possible, while incorporating as 
many as possible of the sophisticated STS concepts developed in VOSTS. 

The design process was divided into four phases: 

1) Initial consultation and design of English draft version 

2) Focus testing and revision 

3) Pilot test of revised English version 

4) Revision, translation into French, and final test in both official languages 
These four phases are described separately in the following sections. 

4.2. Initial Consultation and Design 

A small Advisory Group of science consultants and teachers was recruited for the initial 
consultations. 24 The group was asked to examine the original VOSTS items, and advise 
the project team on whether or not the items were appropriate for the target group of 
Grades 10 and 12 students. The unanim ous conclusion was that the items could not be 
used as is, and would have to be simplified. Some felt that only the advanced level grade 
12. students would be able to handle the VOSTS items. All agreed that Grade 10 students 
could not be expected to deal with them. 



It was also recognized by everyone - client, project team and educational advisors - that 
some of the issues covered in VOSTS are of marginal interest to grade 10 students. 
Nonetheless, it was agreed that we should try to design a preliminary instrument that 
built on these issues, since the VOSTS instrument has already generated a national 
database at the grade 12/OAC level with which we could compare results. It was also 
recognized, however, that our initial goal might not be achievable. 



The VOSTS instrument contains over 100 separate items. It was decided to select about 
one quarter of these items, focusing on topics that most directly relate to the aims of the 
Ontario science curriculum. 25 A detailed comparison revealed that the VOSTS areas 2 
and 8 are not covered by the Ontario aims, which focus more on the nature of science and 
technology and their effect on the individual and society. The social construction of 
scientific ideas and processes, and the nature of the scientific community, are important 
domains in STS, but are not covered in the aims. 




-ll- 



lg 



A meeting between the Project Team, Ministry officials and the Advisory Group was held 
to finalize the selection of VOSTS items that would be adapted. " Table 2 indicates' the 
VOSTS item numbers that were selected. 

The next step was to adapt these items to a form that would be easier to read. Exhibit 1 
shows an early design for item 40211, that was ultimately rejected (see Appendix B). It is 
instructive, however, because it clearly illustrates a key design issue. A typical VOSTS 
item presents a list of responses, in a specific order, usually from agreement down to 
disagreement (sometimes the order is reversed). There is a good possibility that students 
with poor reading ability will tend not to read all the options before choosing. 

To address this concern, the left-hand column in Exhibit 1 (see Appendix B) was designed 
to tell the student quickly that the options are listed in a specific order running from 
agreement with the initial statement, to disagreement. Then they would only have to 
read the detailed options relevant to their initial reaction in favour of or against the 
initial 
statement. 

Table 2 

VOSTS Item Numbers Selected for Adaptation 



10111 


40211 


50211 


60111 


70121 


80131 


90211 


10211 


40311 


50313 


60221 


70221 




90411 


10411 


40412 




60511 


70711 




90621 


10421 


40421 




60611 






90641 




40451 










90651 




40511 










90711 

90811 



While this design was ultimately rejected, it incorporated some important features, which 
were re tain ed in the preliminary test version. Students completing VOSTS items are 
ins tructed to read the top statement and think whether or not they agree with it. Then 
they must read all the options carefully, and select the one that most closely fits their 
opinion. The design in Exhibit 1 makes the initial reaction explicit, and asks the student 
to select an option. Then they must decide, using the more detailed descriptions, on how 
to fine time their response. This feature appears in a different way in the preliminary 
test version. (However, by the final version the opening statement had been eliminated.) 

An "I don’t care" option was added, to see if it elicited significant responses, compared 
with the "I don’t know" response. The open-ended question at the end allows students to 
explain what they mean, or why they don’t understand, or add a nuance or option that 
was not covered. Both features were retained in the preliminary test version, and only 
the open-ended question survived in the final instrument. 

The form ul tima tely selected for prehminary testing is displayed in Exhibit 2, next to the 
original VOSTS format (see Appendix B). There are several features to note: 

the initial statement is presented clearly, in large capital letters, in a box 

the first question allows students to record their immediate reaction to the 



statement, including whether or not they even care about the subject 

• the second question presents options, adapted from the original VOSTS item 

* the options are not necessarily ordered sequentially as in the VOSTS items 
the language is simplified 

the third open-ended question allows students to write in a response in their 
own words 

In addition to adapting a selection of VOSTS items as described above, a battery of 
bipolar statements was drafted. These statements were designed to measure attitudes 
toward science, technology and STS-related issues. They were not designed to measure 
students’ comprehension of STS issues, only what they feel about them. There were 
several reasons for including this component in the preliminary instrument: 

* it allows attitudes to be measured, in addition to cognitive beliefs 

* it was desirable to compare this simpler format with the more complex 
format of the adapted VOSTS items 

* the bipolar statements could cover issues that the longer VOSTS adaptations 
could not 

The preliminary instrument that emerged from this design phase consisted of two 
elements: 

1) Twenty-six "knowledge" statements, adapted from VOSTS items 

2) Thirty-seven "attitudinal" statements 



4.3. Focus Testing 

The preliminary instrument was tested separately with teachers and with students. Both 
sessions were conducted at the offices of Market Facts of Canada, 77 Bloor St. West, 
Toronto. Focus groups were conducted by Ms. Jean de Boerr of Market Facts. Project 
Team members and Ministry officials observed from behind a one-way mirror. The 
subjects were told at the beginning that they were being observed. 

Teachers 

The session was held from 6:00 p.m. to 9:00 p.m., Wednesday, 20 September 1989. 11 
teachers were recruited from the Toronto area. A cross section of disciplines and program 
levels (basic, general, advanced) was represented. 

The teachers were give the first part of the instrument, consisting of the adapted VOSTS 
statements, and asked to fill in the questionnaire. They were then given the bipolars to 
fill in. 

A discussion of the two parts of the instrument was led by Ms. Jean de Boerr of Market 
Facts. The rest of the Project Team were behind the one-way-glass. After an hour of 
discussion, they came into the room to continue the discussion with the teachers. 



Students 



The session was held from 5:00 p.m. to 8:00 p.m., Thursday, 21 September 1989. 

Students were recruited from schools in the Metro Toronto area. The breakdown of grade, 
level and gender are given in Table 3. 



Table 3 

Student Focus Group 
Grades, Levels and Gender 

Grade 10 Grade 12 OAC 

B G A B G A 

6 11 11 1 

2 1 

Each student was given four of the adapted VOSTS items, and all of the attitudinal 
statements. This ensured that students had adequate time to fill in their own 
questionnaire. Each of the adapted VOSTS items had two or more students filling it out. 

4.4. S ummar y of Main Findings of Focus Groups 

4.4.1. Teachers 

The teachers who participated in the focus group had a great deal to say about the 
instrument. Even those who were teaching at the advanced level felt that the multiple 
choice items were too complex. Those teaching grade 10, or at either grade at the basic or 
general level, felt that their students would not understand most of the statements. One 
teacher expressed concern, because about 25% of this teacher’s students were also 
studying English as a second language. 

Not only was reading level a concern, but also level of interest. Many of the teachers in 
the focus group warned that the issues covered in VOSTS are not of interest to grade 10 
students, especially students in basic and general level programs. They felt that the 
younger students would simply not be interested in many of the items. This was borne 
out during our student focus group, which included several students from the grade 10 
basic program. They were simply not interested in, or could not comprehend, many of the 
statements. 

When it came to the bipolar statements, most teachers felt that they would be understood 
and accepted by their students. They recommended that even if the multiple choice (or 
some simplified version) was used, the bipolars should remain in the instrument. 

The teachers’ response to the overall exercise was quite interesting. Some were skeptical 
about the STS approach, and were not happy about being judged on their abilities to 
teach a subject that they did not think appropriate for science class. Even among those 
who supported the STS approach, many teachers felt inadequate about teaching it since 
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they were not trained in the area themselves. Another fear expressed by some was that 
the addition of STS topics would overload their teaching responsibilities. "We have more 
than enough to do to cover the required science content, let alone STS”, remarked one 
teacher. 

However, most teachers praised the Ministry for commissioning the present study to 
design an instrument for monitoring students’ attitudes and understanding of STS issues. 
Some teachers urged that the instrument, whatever it turns out to be, should be used for 
teacher inservice, and by teachers in the classroom to explore STS issues with their 
students. This point was also made quite emphatically by several members of the 
Advisory Board. 

In summary, teachers recommended against using the multiple-choice items as presented. 
They recommended radical simplification if we wanted to administer the questionnaire to 
the younger students, and to those with lower reading abilities. They also recommended 
keeping the bipolar items. 

4.4J2. Students 

A number of students in grade 10 basic level participated in this group. In the early part 
of the discussion, these students (most of whom were female) were noticeably reticent to 
discuss the instrument. As the discussion progressed, they admitted that they were not 
very interested in many of the statements. In contrast, some of the older students, even a 
few at general level, were more open to discuss some of the topics. In general, the 
assessment of the multiple-choice items was that they were too difficult. 

The open-ended question on the multiple-choice items proved to be very useful in this 
regard. Many of the students took the time to fill in the box at the end of the multiple 
choice items. In a number of cases, the student wrote that they did not understand the 
item, yet they circled options (other than "I don’t know") for both questions 1 and 2. This 
highlighted the need for precision and careful design. To these students, clearly "I don’t 
know" was not synonymous with "I don’t understand". 

When it came to the bipolar items, the students responded much more favourably. They 
all took the time to answer them (whereas some of the students in grade 10 basic level 
did not answer their multiple-choices). During the discussion, they recommended that the 
bipolar statements be included, because they were interesting to fill out, and easy to 
understand. 

4.4.3. Revisions 

Based on the discussions and the feedback from the teachers and students, as well as 
further consultations with the Advisory Group, mqjor revisions were made to the adapted 
VOSTS items. The initial statement was dropped, as well as the initial question asking 
students to respond to the opening statement. Each item was edited down to three or 
four choices. The result was a more conventional, much simpler, multiple-choice 
questionnaire. 
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The main advantage of the change was the reduction of the amount of reading. Instead of 
having one item per page, several items could be placed on one page. This reduced the 
size of the questionnaire, making it less intimidating. The main disadvantage, of course, 
is the loss of the subtler shades of meaning which characterize a VOSTS item. However, 
all those involved in the project agreed that the resulting instrument would allow 
comparisons to be made between levels within one grade, and between grades 10 and 12. 

The open-ended question at the end of each multiple-choice item was retained for the pilot 
test. The bipolars were kept as well. 

There is a total of 38 multiple choice items, presented six on a page, and 44 bipolar 
statements. The instrument was pilot tested province-wide in both urban and rural areas, 
in the English language. 

4.5. Pilot Test 

The pilot test had a number of specific objectives: 

• to gain insight into the level of acceptance by authorities at various levels 

• to test the degree of ease with which students in urban and rural settings 
would be able to answer the questionnaire 

• to test the appropriateness of the language level of the questionnaire for 
grade 10 students 

• to test the comprehension and coverage of the multiple-choice statements 

• to solicit open-ended responses (in order to test comprehension and adequacy 
of coverage of issues in the multiple-choice statements) 

• to test the attitudinal statements 

• to test one OAC class in order to provide a comparison with the same level of 
sophistication used to develop the original VOSTS items 

During the test setup we continued to encounter problems with the grade 10 population, 
and the basic grade 12. As described below, we could not get approval to test students in 
basic level programs at either grade, nor students in general level programs in grade 12. 
Nonetheless, we tested all other subgroups, and were able to generate useful data. 

4.5.1. Setup Procedures 

Since this test required in-class time of students, we obtained permission both from the 
Board of Education and the school principal. 

In addition, a letter was drafted for signature by the principal, to ensure that appropriate 
consent forms had been collected. This procedure was adopted to satisfy the requirements 
of the Freedom of Information and Protection of Privacy Act. 

4.5.2. Site Selection 

The questi onnair e was made slightly longer than the final version would be, in order to 
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cover as much content as possible and to provide some redundancy. 27 As a consequence, 
the test could be administered only in class periods of more than one hour duration. For 
this reason, site selection was limited to schools that had at least one class period of 75 
minutes, or two consecutive science periods on any one day of the week.. This requirement 
limited the number of sites that were eligible for the test. 

In order to select appropriate sites, we first had to determine the overall distribution of 
secondary schools in Ontario. In particular, we needed to know the representation of the 
various sub-populations that needed to be studied (basic, general, advanced, urban, rural, 
public, separate), in order to ensure that we did not use too mainy schools with 
sub-populations that are rare. 28 When selecting sites for the final test, we would not be 
able to choose ones used for the pilot test. So it was essential to determine whether or 
not any of the sub-populations were under-represented. 

There is a total of about 600 English language Ontario schools (public and separate) that 
offer science courses at both grades 10 and 12. Using this as the total school population, 
we then determined the breakdown according to levels within each grade. We restricted 
the population to schools whose class sizes are 20 students or more (numbering about 560 
in total). 



Table 4 

Number of schools offering science courses 
with 20 or more students 



Grade 10 


Basic 


<80 




General 


>450 




Advanced 


>450 


Grade 12 


Basic 


<20 




General 


<250 




Advanced 


>450 



Table 4 indicates that grade 12 basic level science programs represent a small percentage 
of the total n vim her of students enrolled in the subject, and that even in grade 10 
programs, the number of basic level science courses offered is low, relative to the other 
two levels. Furthermore, in grade 12 the general level is considerably smaller than the 
advanced. 

In light of these findings, the sample selection for the preliminary test was modified 
slightly. It was decided to try to test all three levels (basic, general, advanced) in grade 
10, but not in grade 12. This would allow the widest possible test of reading level at 
grade 10, while saving the few schools offering Basic Grade 12 Science for the final test. 

As for students in Basic Grade 10 programs, prospective sites were selected and informal 
inquiries were made. In two cases - one' small rural board and one large urban board - 
initial indications at the school level were positive, but later formal requests at the Board 
level were denied. In the rural case, there was interest in participating in the fin a l test. 
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but not the preliminary one. At the urban Board, it was felt that the questionnaire was 
too complex for students in Basic Grade 10 programs. 

Given the urban Board’s reaction regarding the inappropriateness of the questionnaire for 
students in Basic Grade 10 programs, it was decided not to continue searching for basic 
classes for the preliminary test. Since the population of basic level programs relative to 
the others is so low, it was felt that basic programs should be saved for the final test. 
Table 5 gives a breakdown of the pilot test sites and the student populations at each site. 
The top row is a rural school. All the rest of the sites are in urban locales. 



Table 5 

Test Sites and Sub-Populations 
(Total Population Size: 215) 



Site 


Grade 10 Classes 


Grade 12 Classes 


Location 


Basic General 


Advanced 


Basic General Advanced 


Dundas 


19 


23 


15 


Ottawa 


31 


22 


18 


North York 


18 


20 


11 


TOTAL 


68 


65 


44 



The total student population tested was 215 students, which is a very large sample for 
any pilot test. The results were useful for comparing general and advanced within grade 
10, and advanced grade 10, advanced grade 12 and OAC. 29 

4.6. Summary of Main Findings of Pilot Test 

When we compared the percentages of write-ins for the first half of the questions with the 
percentage for the second half, it was obvious that the numbers of students writing in 
their own comments dropped dramatically. This drop can be attributed to a fatigue 
factor. Clearly students were becoming tired or losing interest long before the end. This 
possibility was predicted by several of the school authorities who were asked to approve 
testing. In fact, some did not want to subject their students to such a lengthy 
questionnaire. 
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This problem was dealt with in the next round of testing, by cutting the total number of 
multiple-choice questions in half. Each set of students was given only one half to answer. 
Within each half, two sets were prepared in reverse order, so that when the scores were 
totalled, any remaining fatigue factor could be averaged out. 

Question #38 received an anomalously high percentage of write-ins. When these were 
read, we found that students were reacting very strongly against the notion of having to 
take more courses, whatever they might be. Comments of the sort "students should 
decide themselves what they want to take" were the rule. This evidence indicated that 
the question was inappropriate, and it was dropped. 

Analysis of the write-ins, and of the percentage responses, allowed the project team to 
weed out some spurious items, and to improve others. Table 6 gives a summary of the 
changes made to the multiple-choice items, based upon the pilot test. 

The total number of multiple-choice items in the final instrument dropped to 29. 

In addition to these changes, a number of the bipolars were edited, and several new ones 
were added. The total number of bipolars in the final instrument went up to 49. 



Table 6 



iple-choice # 


Change 


Reason 


2 


edited 


removed bias 




option added 


improved range 


3 


discarded 


one option swamped (92%) 


4 


converted to bipolar 


more appropriate measure 


10 


edited 


improved clarity 


12 


discarded 


problematic, swamped (84%) 


22 


edited 


improved clarity 


26 


discarded 


swamped (83%) 


28 


edited 


simplified 


32 


discarded 


problematic, simplistic 


33 


edited 


improved clarity 


34 


edited 


(steps of the) scientific method 


35 


discarded 


swamped (80%) 


36 


discarded 


problematic, simplistic 


37 


discarded 


problematic 


38 


discarded 


problematic 



4.7. The Issue of Student Reading Level 

Throughout this project, the issue of student reading level was a mqjor concern. During 
the setup phase for the pilot test, as noted above, several officials refused to subject their 
students to such a long questionnaire. Furthermore, as the focus groups indicated earlier, 
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the comprehension level of many grade 10 students is much lower than that of their 
counterparts in grade 12. What might be acceptable for grade 12, might not be useful for 
grade 10. 

Nonetheless, the pilot test indicated that students were generally able to deal with the 
material presented to them. A random analysis of the write-ins showed that even grade 
10 students were able to comment on the issues, albeit at a simpler level than that of the 
older students. 

While the sophistication of the original VOSTS instrument has been eliminated, the topics 
covered are the same. For purposes of comparing grades and other subgroups, the 
resulting instrument will be able to provide extremely useful information. If used for 
teaching purposes, teacher inservice and teacher training, the instrument can improve 
STS education over time. Variations in reading level among students can be 
accommodated by teachers administering the questionnaire. 39 

A major test of the instrument was carried out across the province, in both official 
languages. As described below, the instruments were broken into smaller sections. In 
this way, no one student had to read more than half of the entire "bank" of items. This 
technique assuaged the fears of those officials who had refused to participate in the pilot 
test. The revisions, and the technique of splitting both parts of the questionnaire, sufficed 
to ensure them that the students would be able to handle the assignment. 

The final instrument appears in Appendix A. This version, in both official languages, was 
used to conduct a final province-wide test. 

4.8. The Final Test 

4.8.1. The Questionnaire 

The "h ank " of multiple choice questions and attitudinals (bipolar items) to be included in 
the final stage of the study equalled twenty-nine and forty-nine respectively. The 
desirability of incorporating them into the framework of a single, all encompassing 
questionnaire had to be weighed against the following considerations: 

• The tim e tak en to a dminis ter a draft version of such a unified questionnaire 
during the pilot study was in excess of one hour. The average class period in 
most schools (barring the semestered ones) is, on the other hand, limited to a 45 
minute duration. This necessitated that the questionnaire length be shortened to 
allow for a response time of not more than 35 minutes (the remaining 10 minutes 
being required for other related preliminaries and post-completion procedures). 

• At the time of the pilot study, some school teachers and Board officials expressed 
reservations about the questionnaire length especially in the context of students 
in Grade 10 Basic level programs. 

. The pilot study indicated that - most probably due to respondent fatigue or 
boredom - the quality and quantity of responses began deteriorating somewhat 
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along the length of the instrument, especially in the latter half. 

• The targeted sample size i.e. 1,500 students was large enough, that halving the 
effective sample base by dividing the "question bank" into two independent sets 
would still produce completed questionnaires in the subgroups, large enough for 
independent examination. 

With these considerations in mind, two versions of the final test instrument were 
developed, each having 15-16 multiple choice items and 24-25 bipolar attitudinals. The 
twin instruments thereby arrived at, were ones that could be accommodated within a 45 
minute class period; were not of a length likely to diminish their suitability for certain 
grades/levels (especially grade 10 Basic); and, finally, would not induce respondent fatigue 
due to excessive length. 

Multiple choice items came first in both versions followed by the attitudinal bipolars. The 
knowledge statements "Science is ..." and "Technology is ..." were common to both versions 
as was their position as the "opening" questions. 

The distribution of individual question items between these two independent versions of 
the instrument - which let us refer to as VI and V2 - was done in a way that ensured an 
equivalent division not only of numbers but of subject matter as well. Careful 
intra-version sequencing of items addressed the concern that responses to any one of them 
may bias some of the other responses. 

A final precaution was taken to immunize each of the component multiple choice or 
bipolar items of the two instrument versions from order bias arising as a result of their 
sequential position within the instrument. VI and V2 were each administered in two 
forms, in which the item content remained the same, but their positioning was reversed 
(except for the knowledge statements "Science is ..." and "Technology is ..."; the earlier 
described position of these two items was unchanged.) The four versions of the 
instrument thereby developed were translated into French for the francophone students. 



1. The sample size in total comprised 1,422 students. Each student in the sample 
provided responses to one basic version of the instrument i.e. VI or V2, the sample 
size for each of these versions being 717 and 705 respectively. Only 24 ’completed’ 
questionnaires (or 1.7% of the total) could not be included in the sample because of 
the poor quality of responses. 

2. The sample profile in terms of Grade 10 and Grade 12 science students was as 



4.8.2. The Sample 



follows: 



Version 1 Grade 10 science: 
Grade 12 science: 



386 respondents 
331 respondents 



Version 2: Grade 10 science: 

Grade 12 science: 



384 respondents 
321 respondents 



3. 



Sample selection was done in a three stage process. 



The primary sampling units were secondary schools in the province, having Grade 
10 and 12 science classes, drawn from a list provided by the client. 

In each selected participating school, specific Grade 10 and Grade 12 science classes 
were chosen. 

All students in these classes, who had the necessary consent to take part in the 
survey and who were present on the day the questionnaire was being 
administered, were included in the sample. (Consent forms were required to be 
signed by the parents of students below 18 years of age. Those students who were 18 
years or older could sign these forms themselves.) 

The rationale underlying the choice of the target group (i.e. Grade 10 and 12 science 
students) has already been specified. Other considerations taken into account at each 
stage of the sampling process were as follows: 

a) Selection of schools 

A Grade 10/Grade 12 matched sample, from the same science course level (Basic, ' 
General or Advanced) was to be drawn from each school, thereby preventing the 
effect of inter school differences on grade/level comparisons. 

The sampling frame (of schools) was therefore restricted to those schools having 
both Grade 10 and Grade 12 science classes with the same science course level 
(Basic, General or Advanced) being offered at both grades. 

School data provided by the client revealed that the number of students in any one 
school/grade/science level varied from as few as two to over four hundred. In 
most schools, again, student populations - and class sizes - were higher in Grade 10 
than in Grade 12. Only those schools listed as having at least 20 students in each 
of the target grades/levels were included in the sampling frame. This was done to 
ensure a reasonable student "yield" per sample school; as equitable a distribution as 
possible of the student samples among selected schools and finally, uniformity in the 
representation of Grade 10 and Grade 12 students. 

Apart from Grade and science course level, data were also to be analyzed by: 

Urban/rural students 
Public and separate students 
- Anglophones and francophones 
Central and other regions 

The selection of schools was done to facilitate such analyses. 

• It was expected that from each selected school, the student sample -inclusive of both 
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Grades • would be 30. To ensure the required total student sample of 1,500 an 
initial draw of 54 schools was made. 

b) Selection of classes 

The selection of specific classes (within a grade/level) in the participating schools 
was done primarily with a view to ensuring as close a numerical match as possible 
between Grade 10 and 12 student samples within a school, and in turn equivalence 
in the Grades 10 and 12 subsamples overall. 

In schools having more than one science course level in the two relevant grades, 
required subsamples at the different levels was a further consideration taken into 
account at this stage. 

c) Selection of students 

Once the school and specific classes were selected, no further screening of the 
students themselves took place. Rather, as mentioned above, all consenting 
students who were present on the day the study was administered, were included in 
the sample. 

4. Selection procedure 

a) The starting point was the list of secondary schools having both Grade 10 and 12 
science classes, provided by the Ministry. Of these, the "qualifying" ones made up 
the sampling frame, from which were excluded the three pilot-test site schools (see 
Exhibits 3 and 4 in Appendix B). 

b) Independent samples of English and French schools were drawn. Since they were 
listed by region and within region by public and separate units, a random selection 
ensured proportionate representation for the two samples, as well as for population 
density sub-cells within the two language groups. 

c) The draw of schools at this initial stage of the sampling process, equalled fifty-four 
(see Exhibit 5 in Appendix B). At an estimated yield of over 30 students per school, 
this n umb er was in excess of estimated requirements, having made allowance for 
attrition. This was just as well, for attrition due to various reasons did, as 
expected, erode into the number of such primary sampling units (schools) from which 
the final student sample could be selected. 

Some of the causes of attrition were: 

• Some Boards of Education declined to participate, some did not receive 
official, written authorization from the Ministry in time and would not 
proceed without; some were unable to give the "green signal" on time because 
of survey authorization- procedures e.g. screening by a Research Review 
Committee which sometimes was not scheduled to meet until two months 
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after the initial contact was established, and some Boards had other research 
priorities at the selected schools. 

* At the school level also, some declined to participate due to other survey 
commitments; an inability to accommodate the STS study during the 
scheduled in-field time period; other commitments such as tests or 
examinations, and finally, smaller student numbers than those in the initial 
sampling frame information. 

• A further disqualification factor was the classification of chemistry as a 
Grade 11 credit and that of physics as a Grade 12 credit. Due to 
internal scheduling, Grade 12 students in one school were taking the 
chemistry course and those in Grade 11 were taking physics. Which of them 
would qualify as a Grade 12 science student? Rather than attempting to 
answer the question, this school was excluded altogether from the sample. 

d) As for the "consenting” schools, four of the eleven French schools able to participate 
in the study, had switched from public to separate school boards within the last 
year. This adjustment has been made in all Exhibit tables. The number of other 
such shifts that may have taken place is however not known. The public vs. 
separate distribution of French schools as shown in Exhibit 3 may not therefore 
entirely reflect the currently prevailing patterns. 

\ 

e) On establishing contact with the selected schools, it was also revealed that: i) the 
actual number of students available in each school differed - substantially in some 
cases • from initial estimates; ii) as mentioned above, there was substantial 
inter-school variation in class sizes as well. Logistical and administrative reasons 
precluded the inclusion of "fractions" of a class. Student sample availability in each 
school was therefore also a function of class sizes amongst other factors. 

f) In all, in excess of 2,000 questionnaires were dispatched to a total of 37 schools, 
which it was felt would ensure an end sample of well over 1,500 completed 
questionnaires even after accounting for absenteeism, lack of the required consents, 
and non participation for any other reasons . The completed questionnaires from 
one school were received too late for inclusion in the sample. Data regarding this 
school have been omitted from Exhibits 6 and 7. Exhibit 8 shows the final students 
samples obtained, based on returned questionnaires. 

4.8.3. Fieldwork Organization and Logistics 

Organizationally, the field work stage of the study required intensive interaction and 
coordination with Board of Education and school personnel. The response and degree of 
cooperation received at both Board and school levels was positive. 

The activity flow from sample design to receipt of completed questio nn a ir es from 
participating schools, is given below. 
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1. The first step was to obtain the consent of the relevant Boards of Education, done 
prior to attempting to establish contact with the sample schools. This began during 
the third week of January, 1990. 

2. Board personnel were contacted by telephone, followed by a letter applying for the 
required permission. The procedural aspects for obtaining this varied from one 
Board to another and ranged from immediately giving the go-ahead over the 
telephone itself, to requiring that special submissions for review by committee be 
made. All such specifications were complied with. 

3. The next step involved establishing contact with the school principals • again done 
both by telephone and by letter. This stage was initiated during the last week of 
January, 1990. In those schools that agreed to participate, specific Grade 10 and 12 
classes to be included were arrived at in consonance with school authorities - usually 
the principal, vice principal or science department head. The exact survey timing - 
within a three to four week period - was left to the convenience of the school 
authorities. The field closing date was in fact extended by two weeks to 
accommodate two schools that were unable to conduct the survey earlier. 

4. Sample consent forms (to be signed by parents, in the case of students below 18 
years of age, and by the students themselves for those 18 years or older) were sent 
to these schools along with a confirmation of the classes to be included for the STS 
study. These forms were to be used by the participating schools to collect the 
required consents, prior to the actual ’fielding’ of the instrument in the selected 
Grade 10 and 12 science classes. 

5. Since the questionnaire was to be administered during a regular class period with 
the appropriate class teacher invigilating, all measures were taken to lighten their 
load as much as possible. As mentioned above, sample parent consent forms - or the 
required number of copies thereof where requested - were provided to all schools. 
Individual sets of questionnaires and envelopes for each participating class in a 
school were packed and labelled prior to their being dispatched to the school. In 
each set, the different versions of the questionnaire were pre-arranged in rotating 
order as a "built-in mechanism" to ensure their equitable distribution among 
participating students. Each copy of the questionnaire had completion instructions 
addressed to the respondent (i.e. the student in this case). Instructions for 
administering the data collection procedure were also provided separately, for the 
benefit of the school staff responsible for doing so. (See Exhibit 9 ) Intensive 
telephone contact was simultaneously maintained with the test facilitators in each 
school, to tackle questions and provide clarifications as they arose. 

6. A courier company was used for the delivery of all materials both to and from 
participating schools. 

7. The survey was in field during March and the first week of April of 1990. 
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5. FOOTNOTES 

1. As well as in most industrialized countries. 

2. Einseidel, Edna, Scientific Literacy: A Survey of Adult Canadians (Calgary, 
University of Calgary Program in Communication Studies, 1989). 

3. Ibid. For the U.S. case, see Jon Miller, "Scientific Literacy", Paper presented to 
A.A.A.S., January 1989, San Francisco California. For the U.K. case, see John 
Durant, G.A Evans and G.P Thomas, "The Public Understanding of Science, Nature 
340 , No. 6, July 1989. 

4. Lynne Ainsworth, "Students shun science for arts," Toronto Star. 11 July 1990. 

5. A recent Report of the Premier’s Council on Science & Technology recommends that 
science be made compulsory up to Grade 12; however the report admits that S&T 
are presently made too boring for students, and they therefore recommend that the 
compulsory grade be initially set at Grade 10. 

6. International Association for the Evaluation of Educational Achievement. Science 
Achievement in 17 Countries. A Preliminary Report (New York, 1988); Educational 
Testing Service. A World of Differences. An International Assessment of 
Mathematics and Science. (Princeton, 1989). See also, Newsweek Magazine, 'Not 
Just for Nerds, How to Teach Science to Our Kids," 9 April 1990. 

7. The federal government, for example, established a Program for Public Awareness of 
Science and Technology. It includes two separate components: a publicity campaign 
(including public advertising in print and radio, and contributions to provincial 
education organizations) and a granting program called "Science Culture Canada ", 
which gives grants to organizations and individuals involved in public awareness 
activities and projects. Several provinces have their own public awareness programs 
as well, notably Alberta and B.C. 

8. The National Advisory Board on Science and Technology advises the Prime Minister, 
and each province has its own advisory group. In Ontario, the Premier’s Council on . 
Science and Technology advises the premier. Provincial advisory councils meet 
together as well. 

9. Science For Every Student (Ottawa: Science Council of Canada, 1984). 

10. It is interesting to note, however, that there are a few important areas of STS that 
are not covered by the aims of the Ontario science curriculum. These areas deal 
with the infl uence of society on science and technology, and the social construction of 
scientific knowledge and of technology. The aims of the Ontario science curriculum 
are heavily weighted toward epistemology and the influence of S&T on society. See 
Section 4.1.1, where the aims of the Ontario curriculum and the STS areas covered 
by VOSTS are compared. 

11. Science Curriculum Guideline.. Part 1: Program Outline and Policy, p. 13. 
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12. Science Curriculum Guideline. Part 1: Program Outline and Policy, p. 22. 

13. Science Curriculum Guideline. Part 1: Program Outline and Policy, pp. 10-12. 

14. The relationship between science and technology is, in fact, more complex than this 
simpler picture of technology as applied science. However, the thrust of this aim is 
for students to realize that the two interact in real life, and that many career 
opportunities stem from this link, whether or not the student wants to pursue 
science itself. 

15. Senior Division Advanced-Level Chemistry: Educators’ Report (Toronto: Ontario 
Ministry of Education, [1988]). Senior Division Advanced-Level Physics: Educators’ 
Report (Toronto: Ontario Ministry of Education, [1988]). 

16. Aikenhead, G.S. Fleming, R.W. and Ryan, A.G., "High school graduates’ beliefs 
about science-technology-society. I. Methods and issues in monitoring students’ 
views," Science Education. 71 (2), (1987), 145-161. Other researchers have produced 
different instruments. The work in STS evaluation has been researched and 
presented in a Literature Review as part of this study. See Part Two 

17. The requirement for a single instrument for both grades and all course levels was to 
allow for comparative studies over time that compare student responses as they 
move through the system. 

18. A future study of considerable interest would be to design an instrument for grade 7 
and use it to compare with grade 9 and 12. This would measure the change in 
views between entry level students into the first year of the intermediate/senior 
program, students entering high school science and students completing high school 
science. 

19. See Appendix C. 

20 Rosenthal, D.B., "Two approaches to STS education," Science Education 73 (5) 
(1989), 581-589. 

21. Test on Understanding Science (Cooley & Klopfer, 1961), Nature of Science Scale 
(Kimball, 1965), Science Process Inventory (Welch, 1966), Test of the Social Aspects 
of Science (Korth, 1968), Nature of Scientific Knowledge Scale (Rubba, 1976), and 
the National Assessment of Educational Progress (Bybee, Harms Ward & Yager, 
1980; Hueftle, Rakow & Welch, 1983). 

22. For example, journals such as Science. Technology & Human Values, or Bulletin or 
Science. Technology & Society, and articles and books such as Aikenhead, G.S. , 
"Collective decision making in the social context of science", Science Education, 
69(4), (1985), 453-475; Barnes, B. & Edge, D., Science in context (Cambridge, 
Mass.: MIT Press, 1982); Bybee, R.W. (Ed.), Science-technology-society. 1985 
NSTA Yearbook (Washington, DC: National Science Teachers’ Association, 1985); 
Fleming, R.W., "Literacy for a technological age", Science Education, 73( 4), (1989), 
391-404; Gauld, C., "The scientific attitude and science education: A critical 
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reappraisal”, Science Education. 66( 1). (1982), 109-121; Snow, R.E., "Core concepts 
for science and technology literacy", Bulletin of Science-Technoloerv-Societv. 7(5/6), 
(1987), 720-729; and Ziman, J., Teaching and learning about science and society 
(Cambridge: Cambridge University Press, 1980) and An introduction to science 
studies: The philosophical and social aspects of science and technology (Cambridge; 
Cambridge University Press, 1984). 

23. The number of topics within each mqjor section, and the number of sections 
themselves, may be expanded in the future. In fact, section three has been left 
blank in order to leave room in the framework for future items emphasizing the area 
of technological literacy. The VOSTS item pool currently emphasizes scientific 
literacy because this area was more fully developed in the STS area when the 
research began in 1984. 

24. To ensure provincial representation, these six individuals included the incumbent 
Chairs of STAO and SCCAO. 

25. Table 1 and Table 2 summarize the Ontario aims and the VOSTS topic areas 
respectively. A more detailed description of the Ontario aims can be found in 
Science Curriculum Guideline. Part 1: Program Outline and Policy, pp. 10-12. 

26. Several schools were surprised by this procedure, as they did not realize the Act was 
in force. 

'• . / ' 

27. For example, several items deal with the same issue, but come at the topic from 
different perspectives. This allows for consistency and comprehension checks. 

28. For example, we found that students in basic grade 12 programs are extremely 
under-represented in Ontario, in both urban and rural regions. 

29. One OAC class was included to provide a comparison to VOSTS data, which was 
designed with graduating high school students. 

30. For example by administering the questionnaire in smaller segments. 
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PART TWO 



A LITERATURE REVIEW ON EVALUATION AND STS ISSUES 

L INTRODUCTION 

The importance of the science-technology-society (STS) theme in science education has 
been widely recognized. The past decade has seen increasing attention to STS being paid 
by science curriculum developers in Canada and other countries (Orpwood and Souque, 
1984; Gaskell, 1982; Bybee, Harms, Ward and Yager, 1980; Bybee, 1987; Harms and 
Yager, 1981; Science for Every Student, 1984). As a result of this new emphasis, the 
Ontario Ministry of Education has incorporated STS issues into its new science 
curriculum guidelines for secondary education (Curriculum Guideline, Program Outline 
and Policy, 1987). In addition, a special Science and Society course for OAC has been 
developed (Part 15 of Curriculum Guideline, 1987). 

The introduction of STS-style teaching in science classes has raised the question of 
evaluation. If we are to assess how effective is the incorporation of STS into the science 
curriculum, how do we go about doing so? Are the instruments that ‘have been designed 
to measure student aptitude in scientific reasoning adequate to do the job for STS? Are 
entirely new kinds of instruments required, or can existing ones be adapted for the 
purpose? As an initial step to answering some of these questions for the Ontario context, 
a survey of the STS evaluation literature has been conducted. While the scholarly 
literature in the general area of STS is vast, the field of STS evaluation is only just 
developing - especially in the area of secondary school science, universities and colleges 
have been dealing with STS since at least the early 1970s, and there is an enormous 
literature in this area. For the purposes of this study, however, we have concentrated on 
work being done at the secondary school level. The following journals have been 
consulted for this survey, and would be useful for anyone interested in STS programs or 
evaluation: 

Science Education 

International Journal of Science Education (formerly the European Journal of 
Science Education) 

School Science and Mathematics 
Journal of Research in Science Teaching 
Studies in Science Education 
British Journal of Educational Psychology 
Educational Research 

Research in Science and Technological Education 
Science and Children 

Bulletin of Science, Technology and Society 
Science Technology and Human Values 

This report presents the results of the literature review in four main, sections: 
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Section One reviews the various definitions of STS that one finds in the literature. These 
definitions are placed on a continuum defined by ones’ views of social action and of the 
importance of scientific reasoning for society and individuals. 

Section Two discusses how and where STS has been implemented in science education. 
Programmes in North America, Europe and the Developing Countries are examined. 

Section Three looks at specific evaluation instruments that have been or are being 
developed for STS. The discussion includes instruments for measuring STS-related skills 
and attitudes (for example scholarly research in psychology or pedagogy), and instruments 
being developed to evaluate specific teaching methods and courses. 

Section Four summarizes some of the main research issues that are being studied, using 
STS evaluation and measurement Instruments. These include scholarly research issues 
as well as specific educational evaluation concerns. 

2. DEFINING SCIENCE/TECHNOLOGY/SOCIETY (STS) 

The goal of traditional science education has been the acquisition of scientific knowledge 
in the form of facts and processes which are largely theoretical. The traditional science 
curriculum focuses on preparing students for higher scientific education with little 
consideration for the everyday uses of science. Such a focus has been appropriate for 
preparing aspiring scientists and engineers for future careers. However, less than ten 
percent of the student population ever goes on to work in scientific/engineering/medical 
professions. The STS theme is seen as a way to address the needs of all pupils. As 
citizens they are involved in a multifaceted society. They are faced with daily decisions 
about food supplies, energy sources, population problems, health issues, environmental 
quality, war and peace, and technological advances. (Hofstein, Scherz and Yager, 1986.) 

STS means different things to different people. In reviewing the scope of STS definition 
the "political” persuasion of each author must be kept in mind. On the left end of the 
"political spectrum" one finds the action oriented, socially motivated researchers and 
curriculum developers. On the right end of this spectrum one finds’ the traditional 
science educators who believe that traditional "basic" science is more important than 
social relevancy. 

Robert Yager (1984), from the University of Iowa, can be placed on the "left" end of the 
spectrum. He is a strong advocate of the STS curriculum. According to Yager: 

; 

...science education is defined, as the discipline concerned with the study of the 
interaction of science and society - i.e., the study of the impact of science upon 
- society as well as the impact of society upon science. Their interdependence 
becomes a reality and the interlocking concept for the discipline. Research in 
science education centres upon this interface, (p.36) 

Hofstein and Yager. (1982) emphasize that "concerned" means that science education must 
respond to social changes, and that various societal issues should be reflected in school 
science programs. The goals of science education must be reformulated, they urge, to 
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include the personal and social dimensions of science that have been ignored for over two 
decades, (p.541) 

In the same "political" spectrum as Yager, one finds Reg Fleming of the University of 
Saskatchewan. According to Fleming (1987) STS education also deals with the sociology 
of knowledge. Fleming states that science should be seen primarily as a social institution 
interconnected to other social institutions such as education, government, and the military 
(p.164). 



At the other end of the spectrum one finds proponents of the traditional science 
curriculum. Kromhout and Good (1983) disagree with the Hofstein and Yager position. 
They generally support "the use of socially relevant problems as motivation for a coherent 
study of fundamental science" (p.647). However, the ultimate aim of science 
education, they insist, must be "the coherent structure which is the heart and soul of the 
scientific method" (p.648). They complain that STS emphases are diluting the sound 
teaching of scientific problem solving: 

The computer, the technology of objectives and multiple guess, modularized 
individual study materials, science history, sociology of science, sociological impacts 
of technology (although each has made valid and valuable contributions to 
education), and even the antiscience of creationism have displaced the efforts to 
teach, through direct personal experience and problem solving, an understanding 0 f 
the power and scope of the problems on which science can cast significant light as 
well as those questions of value judgment on which science does not lead directly to 
a solution, (p.648-649) 

Kromhout and Good believe that any attempt to include socially relevant topics would 
distract science teachers from educating students about the structure and methods of 
science. Furthermore, they assert that use of social issues in curriculum organization is 
dangerous because social activists will manipulate and pervert science education, (p.650) 

Bybee (1987) has taken the middle ground in de finin g STS and science education. Bybee 
sees science as a method conducted by people in a social context. Thus, he advocates 
three objectives for science education: 

• scientific and technological knowledge, 

• scientific processes, 

• personal/social applications. 

Bybee insists that STS education must have cognitive as well as social aims. A number of 
cognitive goals of science education (in the STS context) have been identified, by several 
authors (Yager, 1986; Fleming 1987; Bybee, 1987). These include: 

• the development of reasoning, 

• extension of the inquiry goal to include decision making, 

• concern for the opportunities and requirements needed for a wide variety of 

careers, 

• focusing instruction on exploration rather than "coverage", 

• developing a favourable and accessible image of science and scientists. 
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The role of technology in society, and its relationship to science is considered to be an 
important aspect of STS. Fleming (1986) believes that the relationship between science 
and technology should be viewed as symmetrical. Technology is not applied science. 
Rather than depending on the culture of science, technology has its own separate cultural 
agenda. (See also Hurd, 1986) 

Solomon (1988) believes that technology, in the context of STS courses, will mean neither 
the intricacies of microelectronics and mechanisms, nor those "big machines" that are the 
general public’s equation to technology. Solomon states that students need to see 
technology as the application of knowledge, scientific and other, for social purposes. 
According to Solomon there is a long history of teaching the "applications" of knowledge 
which includes technology as applied science. This tradition assumes that the 
introduction of technology into society is unproblematic. This view, Solomon asserts, is 
contrary to the aim of teaching STS. 

David Layton (1988), from the University of Leeds in the United Kingdom, also regards 
technology as a separate entity from science. According to Layton, technology must be 
recognized as an autonomous co-equal to, and not subordinate branch of, science. Layton 
believes that the analysis of contextual values (social, economic, aesthetic, moral and 
others) should be part of an attempt to understand the interactions between technology 
and society. 

In contrast to the prevalent view that technology and science are distinct, albeit related, 
the Ontario Ministry of Education states in its aims of science education that technology 
must be learned as applied science. Granted this goal also includes understanding the 
interaction between science and technology. (Curriculum Guideline, 1987) 

3. IMPLEMENTING STS IN SCIENCE EDUCATION 

This section explores some of the STS programs that have been implemented in North 
America, Europe, and the Third World. 

Since the late 1970s, at the university level, the involvement of ethics and values in the 
practice of science and technology has become a growing concern of a variety of 
researchers. In 1978, in the U.S., a national survey to determine the extent of academic 
activities in the field of science, technology, and h uman values in colleges and university, 
showed that courses or programs were offered in 919 institutions of higher education 
(American Association for the Advancement of Science, 1978). In addition, courses or 
programs related to STS; environmental concerns; health care; and contemporary moral 
and ethical problems related to science and technology were offered in 879 institutions, in 
the U.S. (Hurd, 1986; McConnell, 1982). However the present report will only deal with 
pre-university STS programs. 

3.1. The United States 

3.1.1. Forces Influencing. STS Implementation 

In the aftermath of the Soviet launching of Sputnik, science education in the United 
States concentrated on producing scientists and engineers through traditional science 

-32- 



0 




39 



teaching methods (Bybee, Harms, Ward, and Yager, 1980). According to Paul Hurd 
(1970), of Stanford University, in the 1950s and 1960s science curriculum designers shied 
away from any direct consideration of the connections between "science, technology, 
society and the individual". Thus the majority of students held negative attitudes toward 
science, science as a career and scientists. 

By the mid - 1970s, science educators and curriculum developers In the U.S. had sounded 
an alarm that science education was in a state of crisis. Hurd (1973) has identified a 
number of factors that were instrumental in developing this perceived crisis. Profound 
changes had occurred in American society and there was increased dissonance in 
American culture. Public disenchantment with science and suspicion of technology 
worked hand-in-hand with a prevailing lack of confidence in schools. The consensus 
emerged that science education in the U.S. was in need of change. 

According to Hurd (1986, 1973, 1970) the key issues in the reform movements of the 
1970s and the 1980s were: 

• the purposes of science education, 

• the context of the science curriculum, 

• the scientific subject matter most worth knowing, 

• the conditions that foster that knowing. 

A National Science Teachers Association (1978) study identified the following factors as 
related to the crisis in science teaching: 

1. decline in student population; 

2. increasing diversity among students; 

3. decline in local and national funding; 

4. loss of public confidence in science and public education; 

5. decline in science course enrolments; 

6. unfavourable student attitudes toward schooling; 

7. science curricula unsuited to current needs; 

8. inappropriate pre-service and post-service teacher education 

9. unionization of teachers; 

10. mandated accountability. (McConnell, 1982) 1 

Project Synthesis (Harms and Yager, 1981), a two-year study of "the countenance of 
science education” supported by the National Science Foundation, resulted in the 
following recommendations: 

1. "A major redefinition and reformulation of goals for science education; a new 
rationale, a new focus, a new statement of purpose are needed. These new goals 
must take into account the fact that students today will soon be operating as 
adults in a society which is even more technologically.oriented than at present; 
they will be participating as citizens in important science-related societal decisions. 
Almost total concern for the academic preparation goal as is currently the case is a 
limiting view of school science". 

2. "A new conceptualization of the science curriculum to meet new goals; redesign of 
courses, course sequences/articulation, and discipline alliances are needed. The new 
curriculum should include components of science not currently defined and/or used 



in schools. Direct student experiences, technology, personal and societal concerns 
should be foci" (Harms and Yager, 1981). 

3.1.2. Specific Programs 

Largely influenced by Project Synthesis, in the early 1980s the National Science Teachers 
Association began its search for formulating a new science curriculum. The Search for 
Excellence program was developed in 1982 (Yager, 1986). 

Criteria (ideal state conditions) were formulated in five categories: science in 
the elementary school, biology, physical science, science as inquiry, and 
science/technology/society (Yager, 1986). In the s umm er of 1982, 50 programs 
(approximately ten in each of the five categories) were selected as national exemplars 
from the 170 identified by the 50 state committees during the spring. 

Some other STS programmes in the United States include: 

SCIENCE IN THE MARKET PLACE in Scotch Plains F an wood High, New Jersey; 

HOUSE HOLD CHEMISTRY AND PHYSICS in Spartanburg High School, South 
Carolina; 

TOPICS IN APPLIED SCIENCE in Jefferson County, Colorado (For more 
examples of STS programs see Hofstein’ and Yager, 1982) 

According to Bybee (1987) in most science programmes the STS component will be greater 
and more intense at higher grade levels. In the U.S., Bybee found that the minimum 
amounts of instructional time being devoted to science-related social issues are the 
following: 

• in elementary school 10 percent; 

• in middle/junior high school 15 percent; 

• in high school 20 percent; 

• in college/university 25 percent. 

3.2 United Kingdom 

3.2.1 Forces Influencing STS Implementation 

According to Joan Soloman (1988) a major force in the launching of STS in the U.K was 
the notion of scientific responsibility as a reaction to the making and the dropping of the 
atomic bomb. (Even before that - in 1939 - the first mqjor STS-style work, J.D. Bernal’s 
Social Function of Science had been published in the U.K This opus established an STS 
tradition in that country.) Solomon states that another force was the new philosophy and 
sociology of science. Soloman claims that the denial of science as value-free knowledge, 
indicates both that the nature of science should be studied, and that the open discussion 
of the social effects of science-based issues should be an objective of education. 



On the other hand Ziman (1980) states that STS in Britain has its roots in rebellion 
against the traditional system of specialization which marks students either as university 




-34- 

41 



bound science student or non-science oriented student. In STS education students are 
seldom being taught specifically for entry to a more advanced level in the same subject, 
and have no reason to pretend that they are working their way up to scientific careers. 
Ziman points out that in this field of education the real needs of the majority of .students 
in each course must be sincerely respected. 

3.2.2. Specific Programs 

The Nuffield Foundation Science Teaching project was an early effort by the British to 
include social concerns in science curriculum. In 1962 a Parliamentary Question was 
raised in the House of Commons about the steps being taken by the new Curric ulum 
Study Group to improve the teaching of science and mathematics. In reply, the minister 
of education announced the setting up of the Nuffield Foundation Science Teaching 
Project. 

According to Mary Waring (1979), the Nuffield Project represented a major landmark in 
British educational practice. It was the first effort to develop an articulated and 
comprehensive set of tested teaching materials in an attempt to achieve coordinated and 
widespread reform in science and mathematics, in secondary schools. An example of this 
program was the "Nuffield Physics Program". Waring believes that the Nuffield Project 
was based squarely upon a set of assumptions which, in part, reflected a history of science 
education peculiarly "English". 

The Association for Science Education (ASE) is the main professional body of science 
teachers in the U.K. The Association has published two general studies programmes for 
post-16 years-old students in schools and colleges: SCIENCE IN SOCIETY (the first 
British high school STS programme), and SCIENCE IN A SOCIAL CONTEXT (SISCON) 
(Solomon 1988). SISCON was inaugurated in 1971 to foster the study of STS in science 
courses at the university and polytechnic level. The British Science, Technology and 
Society Association, which was formed in 1977, was a direct descendent from SISCON. A 
more recent STS program in the U.K. is SCIENCE AND TECHNOLOGY IN SOCIETY 
(SATIS), implemented in 1986. The SATIS project is for young people in the 14-16 age 
range. In addition a new SATIS project for 16-19 year-olds is in preparation (Hunt, 1988). 

In Great Britain STS programs have mostly been implemented in the senior levels of 
secondary school and post-secondary school. The current version of British 0-level exams 
includes an STS component. 

The United Kingdom has not been the only European country involved in STS education. 
In the Netherlands for example, a STS program called Physics Curriculum Development 
Project (PLON) has been implemented. In this project physics curricula for general 
secondary education has been developed in which STS-aspects have been incorporated 
(Eijkelhof and Lajnse, 1988). 

3.3 Canada 

3.3.1. Forces Influencing STS Implementation 

In Canada an STS curriculum has been developed or is being developed in each of the 
provinces since the early 1980s. The impact of the "crisis" in science education from the 
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1970s to the early 1980s in the U.S. (and the U.K.) was felt in Canada as well. The 
Canadian crisis was the result of cutbacks in support for curriculum projects, research, 
and staffs of science education centres. In contrast to the 1960s where prominence was 
given to science courses, the 1970s and the early 1980s back to the basics movement 
generally excluded science as a basic (Gaskell, 1982). 

The Science Council of Canada has been very influential in identifying the importance of 
STS for science curriculum development. Science for Every Student (1984) was the 
product of a comprehensive study of science education in Canadian schools begun by the 
Council in 1980. The research program, designed by the Council’s Science Education 
Committee in cooperation with every ministry of education and science teachers’ 
associations in Canada, was carried out in each province and territory by some 15 
researchers. Interim research reports, discussion papers and workshop proceedings 
formed the basis for a series of nationwide conferences during which parents and 
students, teachers and administrators, scientists and engineers, and representatives of 
business and labour discussed future directions of science education (for more information 
on the study see the background study in three volumes; Orpwood and Souque, 1984; 
Orpwood and Alam, 1984; and Science Education in Canadian Schools. Volume ni. Case 
Studies of Science Teaching. (1984). 

The Science Council of Canada report Science for Every Student (1984) offered several 
pertinent recommendations to science educators. These included: 

• "Science education must provide a more accurate view of the practice, uses and 
limitations of science. 

• Science education must include study of how science, technology and society 
interact. 

* Students must be taught how Canadians have contributed to science and how 
science has affected Canadian society. 

* Teachers and curriculum planners must evaluate students’ progress towards all 
the goals of science education, not just their learning of scientific content." 

The idea that science should be presented as part of Canadian culture and used to foster 
the creation of a Canadian identity had already been forcefully argued in a discussion 
paper distributed by the Science Council of Canada (Page, 1979). It argued that 
Canadians are ignorant of Canadian accomplishments in science and technology, of the 
way science and technology shape Canadian society and of the need to focus Canadian 
research on specifically Canadian problems. 

3.3.2. Specific Programs 

The STS component in science curriculum has been implemented in two ways: either as a 
specific course or program, or as an STS theme integrated into traditional science courses. 
An early example of a Canadian STS course is SCIENCE: A WAY OF KNOWING created 
by Aikenhead and Fleming in 1975. The grade ten course developed in Saskatchewan, 
touches upon cultural influences on science. It is divided into three sections. In the first 
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section, students investigate some of the different ways their community gains and uses 
knowledge, such as through economics, religion, art, and politics. In the second section, 
science is studied in more depth as one of the ways in which the community is affected by 
science and the ways in which science is affected by the community (Aikenhead and 
Fleming, 1975). 

Another example of an early Canadian STS course is SEPARATION OF SUBSTANCES 
(Roberts, McLeod and Orpwood, 1981). SEPARATION OF SUBSTANCES is built around 
a simulation in which students develop technical advice about water supply treatment for 
a fictitious island community. According to Gaskell (1982) conflict over a curriculum such 
as SEPARATION OF SUBSTANCES is likely to arise between scientists, who see it in 
their interests to foster an image of science as being value free, and political groups 
interested in challenging that image. 

A more current example of an STS program in Canada is the SCIENCE AND 
TECHNOLOGY 11 in British Columbia. Two B.C. Provincial Science Assessments, in 
1978 and 1982, each tested student understanding of science and technology issues. The 
Interpretation Panels for the 1982 Assessment at both the grade 8 and grade 12 levels 
reported that they felt these issues were insufficiently stressed in most teaching. The 
1982 report recommended "that science teachers give more emphasis to teaching the 
practical applications to science knowledge and to using the knowledge in new situations" 
(Science and Technology 11, Curriculum Guide, 1986, P.l). 

This recommendation was instrumental in designing SCIENCE AND TECHNOLOGY 11 
in 1984. In September 1986, this new STS course was implemented and taught on a 
provincial scale to approximately 6,700 grade 11 students. The nature of the ST 11 
classes was reported to be almost evenly divided between "even mix" (academic/non- 
academic students) and "non academic". 

According to the SCIENCE AND TECHNOLOGY 11 curriculum guide, this program 
addresses the need to provide students with the attitudes, knowledge and skills necessary 
to become active, informed decision makers in tomorrow’s world. The evaluation of ST 11 
is discussed below. 

3.4. The Less Developed Countries 

STS education in the non-western world, specifically the Third World, has its roots in 
developmental needs. This factor is important in the way STS is implemented and the 
restrictions that are imposed upon it. According to Vardhini, in Less Developed Countries 
(LDCs) the STS approach is essential in education and in science teaching, where it plays 
a fundamental role in the country’s development process (Vardhini, 1982). 

An example of an STS oriented program in the Third World can be found in Nigeria. The 
science curricula for primary schools that are developed draw inspiration from the 
SCIENCE EDUCATION PROGRAM FOR AFRICA (SEPA). SEPA has influenced the 
development of Integrated Science for Primary School in Nigeria, which places emphasis 
on exploration and investigation of the child’s environment (Adeniyi, 1987). 
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Adeniyi (1987) uses four levels of organization in implementating STS in Nigeria: design 
blue print, construction, implementation, and evaluation. According to him, the design 
stage involves stating broad national objectives without giving details about the feasibility 
of such objectives. In this stage a carefully detailed enumeration of the curriculum 
content and suggested general operational plans are provided. In the construction stage 
specific steps are taken towards production of textbooks, manuals, workbooks, and 
equipment. At the construction stage the "what" and the "how" are stated in clear terms 
and trial testings are conducted to "feel out" the feasibility of the program within the class 
room. The implementation stage deals with teacher-student-material interactions; lesson 
notes, textual materials and laboratory activities are planned and executed in class. 
Attempts are made to organize materials so as to suit prevailing local conditions. The 
evaluation stage dwells primarily on assessing the worth of a program. Unstandardized 
local and standardized national tests are used to evaluate students level of performance. 

In implementating STS in the LDCs in addition to the usual STS objectives such as 
science being learner-centred, teacher assisted, action oriented, project based, and topical, 
Rugumayo (1987) emphasizes that its foundation must be firmly embedded in the socio- 
economic and cultural milieu of each country. Science education in the Third World must 
use the current indigenous techniques, arts, crafts, medicine and agriculture. STS in the 
LDCs should integrate the old and new, the national and international, traditional and 
western medicine, traditional and modem agricultural practices and traditional and 
modem technology. 

According to Rugumayo (1987) science education in the LCDs, should aim to promote 
agricultural development, industrial production, and scientific research, which would lead 
to social development as well as modernization. 

Elstageest (1987) insists that no amount of foreign knowledge would solve the specific 
problems of any nation. Knowledge and insight must develop from within so as to tackle 
the problems which arise from within. Thus, the aim of education in LCDs should be self 
reliance. If self reliance is greatly enhanced by the training in and application of the 
process of science, then it is not difficult to establish a relationship between science and 
development. 

The obstacles facing the LCDs in implementing STS education are many fold. Rugumayo 
(1987) underlines the importance of not alienating scientifically trained youth from the 
rest of the population. This challenge can be met by teaching culturally relevant science, 
and by involving the community in STS education. Rugumayo also believes that many 
scientists in the LCDs have found it difficult to apply what they have learned. Science 
education therefore, should emphasise how to apply scientific knowledge and methods in 
tackling practical problems at individual, local and national levels. 

Adeniyi (1987) notes the problem of teaching science (STS or non-STS education) courses 
in English. Some problems inherent in learning science, mathematics and 
technical/vocational subjects through English include: inadequate knowledge of English, 
inadequate mastery of mother tongue, and a lack of congruecy between English and the 
mother tongue. 
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In s um mary it appears that STS applications in LCDs are just as crucial (if not more so) 
as in the developed countries. The restrictions facing the Third World imposes problems 
in implementing STS education in many countries. 

3.5. Teachers’ Reaction towards STS Implementation 

According to Yager and Penick (1986), to effectively implement any STS program there 
are four criteria that must be met: 

1. Significant involvement of local communities in program development and 
instruction 

2. STS as a curriculum focus 

3. Labs defined as the real world ( unlik e the traditional laboratories which were 
devoted to confirmation rather than investigation) 

4. Focus first on qualitative considerations and later on quantitative ones (p.7). 

Furthermore, the authors stress that the science teachers, themselves, must be ready to 
t.flkft the responsibility and the initiative for implementing the STS component in their 
science programmes. If science educators are not supportive of STS programmes and 
willing to implement STS oriented courses, STS courses will be ineffectual. Students’ 
attitude toward the program will be adversely affected as well (Yager and Penick, 1986). 

Bybee (1987) has noted that some science teachers are not prepared to implement the STS 
theme. Bybee believes that researchers in science education should provide instructional 
materials, in-service programs, methods textbooks, and summer-school classes that will 
prepare teachers to implement the STS theme. 

In the U. S., Carol Mitchner and Ronald Anderson (1989) examined 14 secondary 
teachers’ perceptions of a model STS curriculum designed to promote scientific literacy. 
They determined how science teachers perceived the STS curriculum, and analyzed the 
influence those perceptions have on their teaching decisions. They also explored 
curricular issues from the frame of reference of teachers, the key factor in successfully 
implementing innovations. 

The result of the study revealed that whether teachers accepted, altered, or rejected the 
course, they shared similar concerns about the development and implementation of this 
multidisciplinary curriculum. These concerns included: 

• concerns over content, 

• discomfort with group work in the classroom, 

• uncertainties about evaluation in the classroom (the problem of testing 
subjectively), 

• frustration about student population (the students were perceived as under- 
achievers), 

• confusion over the teacher’s role (p.352). 

According to Mitchner and Anderson, in the area of curriculum change, the classroom 
teacher is the key figure in determining the success or failure of a new curriculum. 
Mitchner and Anderson believe that: 



the value and belief system that dictated this teacher role was well defined and 
somewhat rigid. It did not allow for new approaches, such as STS curricula, to 
become easily incorporated and internalized into daily practice. This study 
supports the need for significant attention to teachers’ beliefs and value system. 
Many innovations, such as an STS curriculum and the inquiry programs of the 
recent past, are inconsistent with current beliefs and values of teachers. The 
seeker of change is faced with a choice between moderating the extent of such 
innovations or attempting to influence these [traditional] deep-seated beliefs and 
values (p.353). 

Bybee and Mau (1984) conducted a survey of teachers from 41 countries to study the 
reaction of these educators toward science and global problems. They found that the 
majority of participants in the study supported STS themes in science education and were 
willing to experiment with new STS programs. There were some reservations, however, 
about how prepared they felt to implement the new programmes. 

4. THE INSTRUMENTS AND THE METHODS USED IN EVALUATING STS 

Traditionally the instruments used to access science performance have evaluated scientific 
knowledge or/and cognitive dimensions. Some of these instruments can be adapted to 
evaluate the STS component in science education. It has been only recently that specific 
instruments have been designed to measure students attitudes toward STS. In this 
section, the limitations and potentials of certain instruments and the results of certain 
tests will be explored. 

4.1. Terminology 

In designing evaluative instruments several factors have to be taken into consideration to 
check the potentials and limitations of evaluation instruments (Harty and Beale, 1984). 

Test-retest reliability is the degree to which an instrument’s total score is consistent over 
time. It is used as an estimate of reliability in cases where an instrument is administered 
to the same group of individuals on two occassions. Reliability testing is appropriate 
when it is likely that subjects taking the instrument the second time will not remember 
responses made the first time. The coefficient after correlating the two sets of scores is 
often called "coefficient stability". 

Predictive validity is criterion-related validity. It is concerned with the degree to which 
predictions made by a developing instrument’s overall score are confirmed later by a 
performance index of the same subjects. It also relates to predictions of how well 
individuals will do in a future situation. 

Construct validity is subjective appraisal of what the construct of an instrument 
measures. Construct validity is established by determining the relevancy of the substance 
of an instrument’s items. It is interpreted as the degree to which an instrument appears 
to measure what it purports to measure. 
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4.2. The Domains of Evaluation in STS Education 



According to Yager and McCormack (1989) certain domains should be taught in science 
education and the effectiveness of these domains must be tested. 

The information domain is explained as knowing and understanding which includes: facts, 
information, laws (principle), existing explanations, and theories being used by scientists, 
and internalized knowledge which can be used. 

How scientists think and work portrays another aspect of science. Science education in 
the process domain includes: observing, classifying, measuring, comm uni cating, 
predicting, hypothesizing, testing, identifying, interpreting data, and constructing 
instruments, and physical models. 

The creativity domain in STS education include: visualizing (producing mental images), 
combining objects, solving problems, fantasizing, designing devices and machines, 
producing unusual ideas, identifying, isolating, merging, diverging, converging. 

The current social and political institutions, environmental and energy problems 
necessitates the inclusion of h uman feelings, values, and decision-making skills in science 
curriculum development and program evaluation. The attitudinal domain includes: 
developing positive attitudes toward science in general and in school, science teachers, 
and science careers; making decisions about personal values, makin g decisions about 
social and environmental issues, and exploring arguments on both sides of an issue. 

In the last domain science is used and applied in students’ eveiyday lives. Some 
dimensions of this application domain are: seeing instances of scientific concepts in 
everyday life experiences; applying learned concepts and skills to everyday technological 
problems; understanding and evaluating mass media reports of scientific developments; 
integrating science with other subjects; becoming involved in community-action projects; 
extending school experience beyond the classroom; and emphasizing the interrelationships 
and interconnectedness of science to other human endeavours (p. 46). 

In designing the traditional science curriculum the information domain has been the 
starting point. In designing STS education, Yager and McCormack (1989) stress the 
importance of starting with the real world. The world of applications and connections acts 
as a pathway to important and valuable facets of scientific knowledge and process. They 
point out that to start with the information domain and to move to the application domain 
is difficult for many students. Such an emphasis encourages many students to 
differentiate between real world science (founded bn personal experiences) and school 
science (based on information and included in textbook and course outline). Most 
educators and curriculum developers would agree, Yager and McCormack claim, that the 
aim of science courses for all students to apply the information and connect facts to 
everyday experiences. However, little instruction is concentrated in the application 
domain; which is the major difference between STS and traditional science instruction. 

Since STS education can be based on these five domains, Yager and McCormack can 
stress that STS evaluation should also assess student growth across multiple grade levels 
and in the five domains. 
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4.3. Evaluation Instruments 



The instruments have been organized into five categories according to the methods they 
use and the domains they measure. The five categories are: 

1. Multiple-choice instruments, 

2. Likert-type instruments, 

3. Essay instruments, 

4. Interview instruments, 

5. Views on Science-Technology-Society (VOSTS). 

4.3.1. Multiple-choice Instruments 

Most of the standardized instruments that were developed in the 1960s to quantify 
student understanding of science were in multiple-choice format. TEST ON 
UNDERSTANDING SCIENCE (TOUS) is an early example of such an instrument. TOUS 
is a multiple-choice instrument developed to measure Harvard-sponsored case studies. It 
is a 60-item, 4-response multiple-choice test. The items are categorized into three 
subscales (Aikenhead, 1973): 

Subscale I. Understandings about the Scientific enterprise (18 items) 

Subscale II. The scientist (18 items) 

Subscale III. Methods and aims of science (24 items) 

An example of the use of multiple-choice is found in a study conducted by William Baird 
and Gary Borich (1987). They examined the connection between integrated science 
process skills (ISPS) and formal reasoning ability (FRA). They assess the importance of 
this correlation toward curriculum design and science teaching. According to the authors 
if one trait can be enhanced through effective teaching, with subsequent improvement in 
the second trait, then efforts to manipulate teaching styles, inquiry activities, and process- 
centered curricula to advance this "driver trait" would be warranted. The study shows a 
high correlation between ISPS, as defined by the American Association for the 
Advancement of Science (AAAS) and FRA as measured by pencil and paper version of the 
Piagetian interview. 

Two independent tests of the AAAS science process skills were used for the study. The 
Test of Integrated Process Skills (TIPS) was produced by researchers at the University of 
Georgia (1980), and the Process Skills of Science Test (PSST) was developed at the 
University of Connecticut (1972). TIPS is a 36-item, 4-response multiple-choice test. The 
PSST is a 48-item, 5-response multiple-choice test. The authors use raw scores, without 
attenuation for guessing. 

In addition, two independent tests of formal reasoning ability were used in the study. 

The Group Assesment of Logical Thinking (GALT) was also developed at the University of 
Georgia (1983). The other test - "Classroom Test of Formal Operations" (CTFO) - was 
produced by Lawson in 1978. The GALT is a 21-item, double-response multiple-choice 
instrument, using line drawings to present Piagetian problem situations. The CTFO is a 
15-item, double-response multiple-choice test which utilizes a videotape presentation of an 
adult male presenting the Piagetian problem and posing the question. To be scored as 
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correct, items on both of these tests must be answered correctly and have a correct reason 
for the answer (double multiple-choice format). The four tests were used on 54 subjects 
from three educational-psychology classes for pre-service elementary-education majors. 

Baird and Borich attribute the high correlation between the GALT and TIPS tests to the 
common philosophical and/or theoretical orientation among test authors. Both of these 
tests were constructed by science educators working within the same research 
environment. Any particular orientation to science teaching outcomes within this 
environment could serve to lessen distinctions between test development projects. An 
interesting secondary finding of this study was the absence of any significant correlation 
between the number of science courses taken and integrated science process skills. 

The limitations of multiple-choice instruments are discussed by Aikenhead (1988). 
Aikenhead notes the consistent and significant overestimation of student understanding of 
biological ideas when students were evaluated by a standardized multiple-choice test. 
Their choices on the test did not necessarily reflect their understanding of biology, an 
understanding which evaluators thought was being expressed (Yarroch, 1986). 

On the other hand Aikenhead is a strong supporter of empirically developed multiple 
choice instruments. In his own work (1988) "student positions" for use in a multiple 
choice instrument were empirically developed by analyzing a large number of student 
paragraphs. The results of his study supported the efficacy of a multiple-choice format for 
evaluating student beliefs, provided that the choices are empirically derived rather than 
being deduced from a philosopher's view of science. According to Aikenhead, the "student 
positions" were written in a style that tried to capture a common argument among 
student responses; they were not written in a traditional multiple-choice style. 

Aikenhead’s instrument, Views on Science-Technology-Society (VOSTS) will be discussed 
in Section 4.3.5. 

4.3.2. Likert-type Instruments 

Likert-type tests are probably the most popular and yet controversial instruments used to 
measure attitudes. Despite their many shortcomings, Likert scales have been used to 
measure the strength of group opinion, and to measure the "intensity" of attitude 
responses. In a Likert-type test students place themselves on an attitude continuum 
anchored by the positions "strongly agree" and strongly disagree" with "undecided" as a 
neutral mid-point. In some cases, the neutral position is omitted, forcing respondents to 
choose a position. This section will explore certain instruments and some cases in which 
they have been used. 

Early Examples of Likert-type Instruments 

Aikenhead (1973) describes some of the early Likert instruments developed in the U. S. 
NATURE OF SCIENCE SCALE (NOSS) was created in 1965 and TEST OF THE SOCIAL 
ASPECTS OF SCIENCE (TSAS) in 1968. The NOSS measures opinions about the nature 
of science. The development, validation and reliability measures of the NOSS were 
carried out with college graduates. Thus the NOSS lacks reliability and validity data 
relevant to high school students. 
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The TSAS deals with "the interaction between science and society and those features 
which are related to the social nature of the scientific enterprise itself (p. 543). There 
are fifty-two statements to which the student responds on a five point scale, from 
"strongly disagree" to "strongly agree". The statements were derived from a model which 
suggested three subscales: 

Subscale I. Interaction among science, technology and society (19 items) 

Subscale II. Social nature of the scientific enterprise (21 items) 

Subscale III. Social and political responsibilities of scientists (12 items) 

Another popular test that was developed in the 1960s was SCIENCE PROCESS 
INVENTORY (SPI). It is a 150 item forced choice 2-scale instrument developed in 1966 
by which scientific knowledge evolves (p. 543). On the basis content, the SPI resembles 
the TOUS subscale III. The validation of SPI followed the following procedures: 

• using the literature, 

• devising a model, 

• employing the judgement of "experts", 

• getting the feedback from preliminary studies, 

• testing the instrument for its ability to distinguish among different groups of 
examinees. 

The SCIENCE ATTITUDE INVENTORY (SAI) (Moore and Sutman, 1970) is an extremely 
popular Likert-type instrument used to measure students attitude toward science. The 
SAI is composed of 60 items arranged in a statement format with a five-point Likert-type 
scale for responses. It has two scales; one consisting of positive items and one consisting 
of negative items. A total attitude score is obtained by subtracting the negative scale 
from the positive scale. 

Hugh Munby (1983) of Queens University has studied the validity of SAI. He was 
interested in the way conceptual analysis may be used to investigate the validity of a 
research instrument such as SAI. Many of the items in SAI which might be thought to 
tap attitudes, Munby claims, can be interpreted quite differently. 

Munby also questions the conceptual validity of the SAI. He is primarily concerned about 
the instrument’s inclusion of both realist and instrumentalist perceptions within each 
subscale. The inclusion of both perceptions, he feels, makes the. SAI more useful across a 
broader ran g e of personality/attitude types, rather than measuring specific attitudes (such 
as scientific attitude). Munby believes that SAI needs reworking before it can be used 
with confidence. 

Munby (1982) also conducted a study that examined the impropriety of "Panel of Judges" 
validation in science attitude scales. According to Munby a test is valid if and only if the 
judgement given by a panel has the same contextual meaning as the subjects. In science 
curriculum evaluation, the ’Panel of Judges" includes researchers, test developers, and 
science educators.- The subjects are children or others less articulate in science than the 
"judges". M unb y believes that there can be no certainty that the panel has the same 
context for interpreting the meaning of test items that the subjects have. 
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To avoid mistakes in context Munby suggests that researchers should interview the 
subjects and ask them the exact meaning of their answer and its context. In this way 
researchers can avoid the potential pitfall of the "doctrine of immaculate perception", by 
which researchers assume that their perception of the item corresponds to students’ 
perceptions (see also Aikenhead 1988). 

According to a study conducted by Gardner (1987) standard methods of attitude 
measurement, such as Likert and semantic differential scales, ignore the posibility of 
ambivalence. In particular, they do not distinguish ambivalence from neutral responses. 
This neglect arises out of the assumption that positive and negative affects towards a 
particular psychological object are bipolar, i.e.: uni-dimensional in opposite directions. 
Gardner, contends that this is an untenable assumption. Conventional items that test 
internal consistency, he claims, are ineffective in checking this assumption. It is possible 
for a scale to be multidimensional and still display apparent internal consistency. 

Gardner concludes that attitude researchers should not rely upon item statistics. They 
should test internal consistency data for evidence of the multi-dimensionality of their 
scales. Factor analysis should be routinely employed to check dimensionality. If 
statements reflecting positive and negative affects lead on independent dimensions, these 
should be scored separately and their relationship with other variables analyzed 
separately (p. 248). 

A cautionary note has been sounded by Hazelwood (1989) in England about data gathered 
using Likert responses without checking the ability equivalence of different groups. He 
gave 4,000 fourth-year students from 23 schools in northern England 24 Likert response 
items in a questionnaire. He found that those in the lowest-ability group consistently 
chose more extreme responses than those in the highest-ability group. Replication gave 
further weight to these results. 

In this study Hazelwood presents evidence to support the hypothesis that the less able 
tend to choose more extreme responses. He points out that, "there are people who say 
’very strongly* to every question". He calls this tendency to use or not to use strong 
adjectives "generalized verbal intensity". He notes that various personality traits 
(inferiority, hypomania, psychophathic-deviate, depression) have been reported as being 
correlates of generalized verbal intensity, but general ability has not been thus recorded. 

Hazelwood suggests that some people resort to "always" and "often" response sets when 
the items are too difficult. He suggests further that one sort of response set is to choose 
extreme responses. For example, he asks whether, perhaps, for some 14-year-olds it is too 
difficult a task to discriminate on a five-point scale. Hazelwood points out that there is 
general agreement that semantic/cognitive discrimination is related to ability. He notes, 
however, that its importance in relation to the validity of much survey data is not so 
generally recognized. 

In another study conducted by Shrigly and Koballa (1984) the emotional intensity of 
Likert type science .attitude statements was measured. The authors found that emotional 
intensity can be used to distinguish the attitude concept from other related psychological 
concepts. They suggest that science educators who design or modify science attitude 
scales should continue using item-total correlations and other quantitative judgements are 
also necessary. In addition they recommend that the frequency distribution of data 
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generated by each statement should he examined for skewness and high percentage of 
neutral responses. Both factors can impair emotional intensity of an item. 

The study concludes that science educators who design or modify Likert-type attitude 
scales should rely on more than psychometric measures in judging the validity of an 
attitude statement. 

Aikenhead (1988) stresses the limitations of Likert- type instruments for the STS domain. 
Noting that Likert-type responses offer only a guess at student beliefs, he believes that 
the chances of an evaluator guessing accurately are remote. Aikenhead believes that "the 
almost ideal, unidimensionality of Likert items lies in sharp contrast with the 
multidimensional aspects of authentic science with its socio-technological context ... 
Therefore, one might have predicted the inadequacy of Likert-type scales for monitoring 
student beliefs about STS topics" (p. 617). 

Despite the limitations listed above, likert-type instruments are still used in measuring 
attitudes. Some examples of evaluation using Likert scales are discussed below. 

Kevin Finson and Larry Enochs (1987) used the SAI to test how visits to science- 
technology museums affect student attitudes toward STS. They examined factors such as 
prior classroom experience with STS, instructional methodology employed by teachers, 
grade level, socioeconomic status, school type (private or public), and student gender. 

Data was collected by a pretest-postest control group design by using study-specific 

questionnaires and the Scientific Attitudes Inventory (SAI). 

• \ 

A teacher questionnaire and a student questionnaire were each developed to assess the 
extent to which STS was included in the classrooms used in the study. The student 
version contained two additional items allowing the researchers to assess students’ socio- 
economic status. Neither questionnaire was designed with the intent to measure 
attitudes. Data from these instruments was obtained to establish the extent of STS 
inclusion in course work as a possible factor infl uencing attitudes toward STS (p. 598). 

The SAI was selected by Finson and Enochs because some of its subscales appeared to 
address STS-types of concerns. Subscale 4 deals with science and technology 
relationships in a value-oriented manner. Subscale 5 focuses on science and societal 
relationships - although more on a cognitive than value-oriented basis. Finson and 
Enochs claim that the cognitive aspects of Subscale 5 are appropriate for use in the 
research, because "much of the focus of the STS theme and programs deal with 
technology, which by its very nature requires cognition" (p. 598). 

The study takes into consideration Munby’s criticism of the SAI’s validity (Munby, 1983). 
Finson and Enochs acknowledge the limitations of attitude measures and the absence of 
adequate methods of determining the validity of attitude scales. With this in mind, they 
assessed the content of items in SAI subscales 4 and 5 and found it to be "appropriate" for 
their study. Results of the study indicated that significant differences in attitudes were 
present between visiting and non-visiting students and between grade levels. 

No sig nif icant differences were found between other factors (socioeconomic status, gender, 
private/public school type, and teacher characteristics). 
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Hofstein, Scherz and Yager (1986) compared students’ attitudes toward science in the U. 

S. and Israel. Students’ attitudes were measured by using the National Assessment of 
Educational Progress (NAEP) instrument which uses a multiple choice format. In 
assessing' student attitudes they used booklet four of the NAEP which uses a Likert scale. 
The students had to select one answer that ranged from very positive to neutral to very 
negative. 

The study showed that: 

• U. S. 13 and 17 year-olds both report that their teachers "really like" science 
more often than do Israeli students; 

• U. S. teachers of 13 and 17 year-olds are more anxious for their students to point 
out their errors than are Israeli teachers; 

• U. S. teachers of both 13 and 17 year-olds are perceived as willing to share 
opinions about science-related societal problems more than are Israeli teachers; 

• U. S. 13 and 17 year old students perceived the science content they study to be 
more immediately useful to them than do Israeli students; 

• Israeli 13 and 17 year-olds more often believe that their science studies will be 
useful in the future than do U. S. students; 

• More U. S. 13 and 17 year-olds report that their science classes are boring than 
do Israeli students; 

• More Israeli 13 and 17 year-olds report liking to go to science classes than do 
U. S. students (p. 28). 

The authors suggest that such a binational study helps with the identification and 
understanding of students’ perception of the science education they experience. 

Differences and similarities in the two countries may provide some explanations for the 
results. According to the authors the discrepancies between students’ attitudes in the 
U. S. and Israel may relate to the differences in the educational systems, science teaching 
programs, and curriculum implementation procedures. They identify some differences 
that might be relevant: 

1. teachers preparation 

2. the centralized educational system in Israel 

3. the Matriculation Examination in Israel. 

Khalili (1987) assessed the effects of cultural differences between Australia and the U. S. 
on STS evaluation. He used the "Test of Science Related Attitudes" (TOSRA), which was 
developed in Australia, to evaluate students in Chicago. TOSRA is a multi-dimensional 
test with a sound conceptual basis. It is a 70 item test in a Lakert-type format. It 
consists of seven scales with 10 items for each. These scales are: 

1. social implications of science, 

2. normality of scientists, 

3. attitude to scientific inquiry, 

4. adoption of^cience, 

5. enjoyment of science lessons, 

6. leisure interest in science, 

7. career interest in science (p. 128). 
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As mentioned before, some instruments have been developed specifically to assess an STS 
program. In assessing the effectiveness of British Columbia’s Science Probe 10 in 1986, 
three Likert-type scales were developed. The "Science in Society Scale", "School Science 
Scale’’, and "Careers in Science" were used. These scales were derived or taken from the . 
1982 B. C. Science Assessment (B. C. Science Assessment, 1986). 

The Science in Society Scale attempts to measure a broad range of subjects. One of them 
is the interrelationships and interdependencies of science, technology, and society. The 
results indicated that the program was well received by teachers, but that they were 
concerned about the lack of lab experiments. Most students and teachers viewed Science 
Probe 10 as more worthwhile than the old program. The results showed that student 
interest and motivation had at leat remained constant, if not improved. The new program 
has tended to increase the amount of student reading from the text book, and discussion 
of science issues and values on society. Science students indicated that the program was 
more relevant and interesting. 

4.3.3. Essay Instruments 

Essay questions and short answer questions have been widely used to measure student 
attitude and knowledge. An example of a testing instrument vising the essay format is 
the British Joint GCE O-level and CSE Examinations pertaining to STS topics. The exam 
contains sections on theory application, technology/society interaction, morality of science, 
science/society interdependence, the role of industry in society, and testing of knowledge. 

A specific topic included in the exam is the Chernobyl accident in the former U. S. S. R. 
Students are asked to give written responses after reading a newspaper article about the 
accident. (Joint GCE O-level and GSE Examinations, 1987). 

The limitations of such essay-type instruments in evaluation has been studied by Blum 
and Azencot (1986). According to them, there is always a problem of unreliability due to 
objective scoring. 

Furthermore, students seem to prefer multiple-choice over essay questions. In their 
study, students were given a choice of essay and multiple-choice items. A large majority 
of students chose the multiple-choice items. This was despite the fact that the various 
answers in the multiple-choice items could give hints to those students who answered 
essay-type questions. Few students chose to answer essay questions, nonetheless. Blum 
and Azencot further show that multiple-choice questions proved to be much more reliable, 
without subjective influence in the scoring of the test. 

Aikenhead (1988) also discusses the deficiencies in using student paragraphs to measure 
attitudes. He points out that graduating high-school students are not necessarily the 
most articulate writers. In his study, some students failed to provide critical information 
in their paragraphs about STS topics. 

In addition, Aikenhead notes that ambiguity can occur when students are uninformed on 
a topic, or when they do not fully understand key terms. An unambiguous paragraph is 
one for which the student and the researcher independently pick the same "student 
position" (the student, based on his or her true belief; the researcher, based on his or her 
reading of the paragraph). An ambiguous paragraph, on the other hand, is one for which 
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there is no such agreement. Thus, Munby’s notion of "immaculate perception" can apply 
to student writing as well (p. 617). 

Despite the limitations of the method, there are some researchers who use this type of 
instrument. In Australia Collis and Davey (1986) developed a set of science items to test 
a variety of intellectual skills deemed important in secondary school science. They 
wanted to assess students’ cognitive skills with regard to four content areas (geology, 
biology, physics and chemistry). 

Colley and Davey used two instruments to conduct their study; Structure of Observed 
Learning Outcome (SOLO), and Superitem. SOLO is a model of evaluation which enables 
a child’s answer to a particular question to be classified by the way in which it is 
structured. 

The Superitem created by Cureton in 1965, is a test item in which sets of questions are 
asked about a particular problem situation. Each group of four questions was devised so 
that they formed a hierarchy of difficulty levels. Nineteen of the items were finally 
accepted as meeting the initial criteria. These items were arranged for group testing to 
enable a validation trial to be carried out. Both the Superitem and the SOLO technique 
use students’ written answers for assesment. Collis and Davey believe that the items 
have construct validity and that the test has a high level of internal consistency. 

4.3.4. Interview Instruments 

Interviews have been used as an instrument for cognitive and attitude measurement. 
Osborne and Gilbert (1980) used interview techniques to investigate concept 
underst anding in science. They interviewed students about the potential and limitations 
of technology in relation to the concepts of "work" and "electric current". The researchers 
presented different instances/examples in order to test students’ understanding of general 
concepts. 

Osborne and Gilbert believe that the potentials for the interview method include: 

• applicability over a wide age-range; 

• advantages over written answers e.g. the subjects cannot ignore a question or 
simply guess (thus testing the commitment of a student to a view); 

• advantages over merely asking a student for a definition. 

The limitations of the method include: 

• the limited choice of specific instances; 

• the order of presentation can, in certain cases, influence the answers given by 
students; 

• the conduct of interviews must encourage dialogue and avoid the interview from 
turning into an oral examination; 

• the interview must.be conducted without cuing and giving information that will 
bias a student’s response. 
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The authors suggest that this method can be used in investigating the following: 

1. egocentric view of the world; 

2. the language of students; 

3. retention of early intuitive ideas despite formal teaching; 

4. orientation and attitudes. The use of concrete examples to supplement words such 
as science, physics, chemistry, biology, may provide information and insights into a 
person’s orientation to science. This may lead to a discussion of what aspects of 
science (as clarified through the instances) the subject enjoys, dislikes, and so on. 

The authors feel that this method has considerable potential, particularly because 
students do not feel that they are being assessed against an externally defined system. 

Finely (1984) argues that using clinical interviews has been difficult when research 
questions require quantitative comparisons. The central difficulty has been to construct 
variables that can be used to quantify and statistically compare the results of interviews, 
while maintaining exactly what student conceptions are associated with each variable. 
Finely presents a technique for constructing variables that, he claims, overcomes this 
difficulty. Each proposition represented in these standard terms can be used as a variable 
to compare quantitatively the knowledge of students as expressed in clinical interviews. 

The results of the study indicate that science educators can utilize clinical interviews for 
research questions that require quantitative answers. According to Finely application of 
the technique to clinical interview data provides fundamentally different information 
about students’ understanding of phenomena than does a typical pencil and paper 
achievement test. 

Finely claims that the results of the analyses of clinical interviews can provide 
independent or complementary descriptions of exactly what correct or incorrect 
propositions students use in describing and explaining natural phenomena and how these 
propositions are interrelated. Thus statistical analysis can provide a coherent detailed 
picture of the students’ conceptions. 

Aikenhead (1988) believes that interviews are essential in clarifying student written 
positions in attitude assessment. By interviewing students, Aikenhead found that 
ambiguous student paragraphs or "positions" could indeed be clarified. Aikenhead points 
out that qualitative data that are derived from interviews can help to establish a firm 
base from which to evaluate the validity of quantitative measures. 

A number of studies have shown that interviews should be used as a supplement or a 
clarifying method with other instruments. Renner, Abraham and Biraie (1985) used 
interviews to study physics students’ perceptions of the exploration phase of the learning 
cycle. MacDonald and Bridgstock (1982) found out through interviews that university 
students’ images of science were relatively mild and always qualified. This result was in 
conflict with similar Iikert-type studies of student perceptions of science. 




-50- 

57 



4.3.5. 



VOSTS 



The last instrument that will be looked at is Glen Aikenhead’s VIEWS ON SCIENCE- 
TECHNOLOGY-SOCIETY (VOSTS). This section will examine the process of developing 
VOSTS. In addition, a brief comparison of VOSTS to four types of response models will 
be examined. Certain examples of the applicability of VOSTS and some limi tations of 
VOSTS will conclude this section. 

The Development of VOSTS Items 

A VOSTS item consists of a statement followed by a number of "student positions". The 
subject is asked to choose one position that closely resembles their own position regarding 
the initial statement. The development of a VOSTS item relies heavily upon student 
paragraph responses and student reactions during interviews. Therefore, VOSTS is 
considered to be an empirically-derived multiple-choice instrument (Aikenhead, Ryan and 
Desautels, 1989). 

There are five steps in developing the VOSTS items. In step one, the students respond to 
two opposing statements using a Likert-type scale. They then explain in written 
paragraphs their reason for their answer. In the second step the evaluator analyzes the 
written paragraphs and finds common arguments or views. These common arguments are 
called "student positions". Together, these positions form a very crude multiple choice 
form called mc.2. In addition, the evaluator selects one of the two statements to become 
the item’s statement. In the next step about ten students, who did not participate in step 
2, respond to the revised VOSTS statement in two ways: first by writing to a paragraph 
response, and then by choosing one of the student positions in mc.2. This is followed by 
an interview to clarify the wording of the multiple-choice items. The choices are revised 
and form mc.3. is developed. In step 4 another group of students work through their 
revised multiple-choice items talking aloud about the choices made. This leads to the 
polishing of the item’s wording for greater precision into form mc.4. In the last step a 
large sample of students responds to the VOSTS item. Student positions in mc.4 which 
receive little or no student response and which do not provide interesting feedback, are 
eliminated. The final revised item is designated mc.5. 

The development of VOSTS items includes all four methods (Likert-type, written 
paragraphs, interviews, and multiple-choice format). 

Aikenhead (1988) has investigated the degree of ambiguity held by four different response 
models used to study student attitudes toward STS topics: Likert-type, written 
paragraph, semi-structured interview, and empirically developed multiple choice (VOSTS). 

To test the ambiguity of these four modes of investigation, Aikenhead chose a medium- 
sized prairie city to conduct his study. Two high schools participated, with students 
populations of 400 and 800. Aikenhead used VOSTS statements to investigate the four 
modes of response. 

Aikenhead asks what information was gained, and what was lost, in each of the four 
modes of student response. Each response mode, he notes, involved trade-offs. "The 
method that required the least amount of effort, the Likert type response, yielding the 
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least accurate-most ambiguous-results" (p. 624). Compared with Likert-type responses, he 
found that argumentative paragraphs substantially clarified student beliefs. 

Nevertheless, ambiguity fingered at approximately the 35% to 50% level because some 
students tended to write incomplete or inarticulate paragraphs. In addition, student 
paragraphs require a fair amount of effort to analyze and categorize on a large scale. 

Aikenhead found that compared with written responses, semistructured interviews 
enhanced the understanding of student views by clarifying many, though not all, of the 
ambiguities found in student paragraphs. Of all four response modes, he notes, the 
interview offers the most lucid and accurate data. Its limitation is simply the time 
needed to gather and analyzed the data. 

Finally, the "student position" choices (the empirically derived, multiple-choice response 
mode) reduced the ambiguity down to the 15% to 20% level. According to Aikenhead, in 
terms of obtaining the most accurate data for the least amount of resources expended, the 
empirically developed, multiple-choice response mode seems to be the most efficient of all 
modes investigated by his study. 

Application of VOSTS 

In 1983, the Second International Science Study of the International Association for the 
Evaluation of Educational Assessment (IEA) initiated a student evaluation project in 
Canada (Aikenhead, 1987; Aikenhead, Fleming and Ryan, 1987; Fleming, 1987; 
Aikenhead, Ryan and Desautels, 1989). Because this assessment received the support of 
the Canadian provinces, it became possible to assess Canadian students’ understanding of 
STS topics (along with more traditional forms of science knowledge of interest to the IEA) 
by surveying a large Canadian national sample. The VOSTS items were used in this 
national assesment. A sample of graduating students was drawn from across Canada as 
part of the IEA study. The sample was selected mostly on the basis of course registration. 

The study showed that TV had far more influence on what students believed about 
science and its social, technological context than did numerous science courses 
(Aikenhead, 1988). Furthermore, it was discovered that high-school graduates harboured 
diverse and contradictory beliefs about scientific knowledge. Student paragraphs reflected 
a belief in certain aspects of authentic science; particularly the nature of classification 
schemes, the tentative nature of knowledge, and the social dimensions of knowledge from 
within the scientific co mmuni ty. On the other issues, however, students seemed to be 
uninformed; for instance, on the nature of scientific knowledge, on the motivations for 
generating knowledge, and scientific method (Aikenhead, Fleming, Ryan, 1987). 

Reg Fleming (1988) used VOSTS to examine university undergraduate science students’ 
views of the nature of the relationship between science, technology and society. Two 
hundred students from a chemistry department in a Canadian university responded to 
items on the VOSTS instrument. Based on these responses, questions were designed, and 
using semi-structured interviews, posed to 30 students. The interviews focused on two 
issues: the nature .of science and the role of science in our society. The VOSTS results 
indicated that the science students’ views are remarkably similar to those of high school 
graduates. The interview results show that scientific facts are the central concern when 
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dealing with the nature of science. When dealing with the role of science in society, 
Fleming found that mission-oriented science clashed with pure research. 

VOSTS has also been used to evaluate a specific STS program. Zoller, Ebenezer, Morley, 
Paras, Sandberg, West, Wolthers, and Tan (1987) used a questionnaire comprised of four 
representative statements from the VOSTS inventory to evaluate British Col umb ia’s 
SCIENCE AND TECHNOLOGY 11. The authors posed the following questions: "Do STS 
courses actually work? Does the reality match expectations as far as the goals of STS 
education are concerned? Did the already implemented STS courses have any meaningful 
impact on students with respect to their views on STS related issues?" (p. 14). 

The authors concluded that: 

• The ST 11 course did have an impact in the "desired" direction on the viewpoints 
of students concerning STS. 

• Some of the prespecified superordinate goals of ST 11 which are typical for STS 
courses have been attained albeit some of the related sub-goals have not been 
achieved. 

• A simple straightforward assessment of the key outcomes of the ST 11 course 
(and other similar courses) is feasible, and the data-base thus obtained is 
meaningful, and very useful as a basis for decision-making and future educational 
policy-making concerning STS courses. 

• Students’ viewpoints/opinions/positions about the STS interactions and closely- 
related issues are (locally) contextually dependent. With respect to some STS 
issues they are gender dependent. Although all the students (both non ST and 
ST 11) are not clear about the different roles that science and technology play in 
society, important goals of the ST 11 course were met by ST 11 students, many of 
whom were described by their teachers as "non-academic". 

• The VOSTS inventory (form CDN me. 4) is an efficient, useful instrument for 
assessing students’ views on STS related issues. Its use for cross-cultural studies 
and for comparisons between STS response profiles should take into consideration 
that these profiles are contextually bound (p. 15). 

On the other hand Aikenhead, Ryan and Desautels (1989) identify four limitations of 
VOSTS: 

1. VOSTS is only one strategy for monitoring students’ reasoned beliefs about STS 
topics; namely, a multiple-choice instrument. 

2. The STS topics included for consideration are limited to those emphasizing student 
cognition rather than attitude. VOSTS focuses on the reasons that the students 
give to justify an opinion, their informed viewpoint, their cognitive beliefs. 

3. Current VOSTS items were developed for high school students, particularly 16 and 
17 year-olds. Its language level may not be appropriate for lower levels. 

4. Teaching science through STS means that STS content is taught in conjunction 
with the normal science discipline content of facts, principles, concepts and 
problem solving skills. 

Thus, VOSTS by itself does not assess and STS course. VOSTS assesses some of the STS 
objectives or goals of an STS course. 
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5. 



ISSUES ON RESEARCH 



A number of issues have emerged in the literature that have attracted researchers 
interested in measuring science attitudes and performance in science education. 

Gardner (1975) reviewed the relationships between attitudes to science and other 
variables under a number of broad headings: other educational variables (such as 
achievement and intelligence); cognitive skills; personality; sex; structural variables (such 
as geographic location, social-economic-stature, and home background); school variables 
(such as "climate" and teacher behaviour); and, curriculum and instructional variables. 
Gardner points out that the categories he used were convenient for reviewing purposes, 
but were clearly not distinct and unrelated. 

This section will examine some of the categories described by Gardner such as: Cognitive 
skills, and Gender. Furthermore, cultural and social factors will be briefly examined. 
These factors will be studied as they relate to students’ attitude and the goals of STS 
education. For example, in studying cognitive skills, the issue of gender and attitudes can 
be interrelated. In addition, a brief examination of a specific research method (meta- 
analysis) will be conducted. 

5.1. Testing Cognitive Skills 

In this section cognitive skills are evaluated in relation to specific sdence course. These 
cognitive skills influence the attittude of students. Research in this area falls into two 
categories: psychological/pedagogical research and evaluation of specific courses of 
learning materials. In the first category, many studies have been conducted on the 
relationship between attitude and cognitive abilities. The following variables have been 
reported to have a positive association with attitude: "scientific ability" (Hodson and 
Freeman, 1983), scores on a Piagetian task (Lawson, Nordlan and De Vito, 1975). The 
following variables, on the other hand, have been reported to have a weak or no 
association: verbal reasoning (Hadden and Johnstone, 1982); and creativity (Starr and 
Nicholl, 1975). The following studies involve evaluating cognitive skills and attitudes in 
the context of specific learning situations. An instrument was developed by Texley and 
Norman (1984) to assess the logical reasoning capacity of adolescents in the content of 
environmental science. The researchers used a combination of clinical interviews and 
multiple-choice group test. The study indicates a number of findings such as: 

1. the establishment of an acceptable level of validity for the instrument; 

2. reasoning level of the subjects was below the theoretical levels predicted for them 
by theory; 

3. reasoning ability in environmental science was lower than in areas associated with 
the physical concepts usually tested. 

Hamrik and Harty (1987), in the U. S., examined the influence of resequencing general 
science content on 6th grade students’ science achievement, attitudes toward science, and 
interest in science. . The order of text book chapters in a general science course was 
changed and presented to a group of students. The resequencing was designed to clarify 
content structure and establish interrelationships among mqjor concepts. A control group 
of students were presented with the original chapter sequence. 
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It was found that the students for whom content structured was clarified through 
resequencing the chapters exhibited significantly higher science achievement, significantly 
greater interest in science than students for whom general science content was not 
resequenced. 

The study utilized two instruments: the Iowa Test of Basic Skills and the Children’s 
Attitudes toward Science Survey. The Iowa Test of Basic Skills is a multiple-choice 
instrument designed in 1982 by Hiemymus, Lindquist and Hoover to measure science 
achievement. The Children’s Attitudes toward Science Survey is a Likert-type instrument 
developed by Harty, Anderson and Enochs (1984) to assess students’ attitudes toward 
science. 

The results showed that learning and retaining through the presence of more inclusive, 
anchoring ideas in the learners structure of knowledge was achieved through 
resequencing content. Furthermore, new knowledge anchored or subsumed into concepts 
already in place, relates to other information so it is more meaningful. This study is an 
example of evaluating a specific teaching technique. 

M. Denny (1986) in England, measured students attitudes toward science practicals. The 
study examined whether students see laboratory work as relevant to their everyday life. 
Denny used three divergent instruments: questionnaires, letter writing and drawing 
assignment. The author states that one problem with the questionnaire was that it was 
unclear to what extent pupils’ responses were made on the basis of "what is" as against on 
the basis of "what ought to be". Perhaps another questionnaire can separate these two 
categories; thereby achieving increased validity. 

The result of the study indicated that students consider science practical work to be useful 
only in the school context. Teachers’ ideas about practical work were found to be similar 
to the students. 

The views propagated by the Nuffield project in Great Britain and subscribed to by many 
science teachers, suggest that the value of science practical work to pupils lies in their 
involvement in personal discovery, in learning about the "scientific method, in developing 
a sense of curiosity, and in acquiring confidence in devising strategies for problem solving 
or lea rning social co-operation through group work. However instead of striving for the 
aims propagated by the Nuffield project, the study showed that students associated 
practical work with: 

1. development of manipulative skills, 

2. relieving boredom and generating interest, and 

3. discovering new things and testing ideas. 

5.2 Gender 

Gardner (1975) began his review of the influence of gender on attitudes by stating: "Sex 
is probably the single most important variable related to pupils’ attitudes to science" (p. 
22). Studies which have reported that males have more positive attitudes to science than 
females include: Ato and Wilkson (1983); Johnson (1981); and Menis (1983). Other . 
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studies, however, have reported no statistically significant sex differences in attitudes to 
science; Selim and Shrigley (1983); and Wareing (1981). 

A factor which needs to be considered is the posibility that sex alone is not a significant 
influence on student attitudes. The following studies have been conducted with regard to 
sex and attitude toward science. 

In comparing male and female students, Dale Baker (1985) in the U. S., correlated the 
factors of attitude toward science, spatial ability, mathematical ability, and the scientific 
personality, as measured by the Myers-Briggs Type Indicator (MBTI). MBTI is based on 
Jung’s typologies of personality. The scale descriptors are described in detail in Baker’s 
article. The author chose MBTI because it has been widely used to assess the scientific 
personality and has reported reliability for a junior high school sample. The scientific 
personality as measured by the MBTI would be introverted (I), intuitive (N), thinking (T), 
and Judging (J). 

Spatial ability was assessed by S2, Cube comparisons, from the Kit of Factor Referenced 
Cognitive Tests (1976). Attitude was assessed by the SAI. 

The results showed that males and females with science grades of A and B were found to 
have several characteristics of the scientific personality, good grades in mathematics, but 
negative attitudes toward science. Males and females with science grades of C and D had 
a, more positive attitude toward science, but poor mathematical and spatial abilities and 
few characteristics of the scientific personality. There were no sex differences except on 
the Thinking/Feeling (TF) scale of the MBTI. As expected by the author, females 
portrayed themselves as preferring the (F) scale, the use of personal values when making 
decisions. Males portrayed themselves as preferring the (T) scale, the use of logical 
analysis when making decisions. 

A study by Linn (1987), in the U. S., explores gender differences by using the National 
Assessment of Educational Progress Science Items (1987). The NAEPS items differ from 
many science knowledge tests in that they allow respondents to choose "I don’t know" 
rather than guessing or leaving items blank. The NAEPS is comprised of four booklets. 
The researcher constructed a "test" out of the cognitive items in each booklet. A total 
score on the test consisting of the number of correct responses was calculated for each 
respondent. 

The author found that significantly more females than males chose the "I don’t know" 
answer. The author considered the possibility that the way students felt about science, 
might be a factor in their acceptance of uncertainty when choosing responses. To test this 
possibility, she used science attitude scales administered as a part of booklet four. 
Attitudes toward science were measured in four categories: 

• liking of science 

• attitudes toward science teachers 

• usefulness of science 

• participation in informal science activities. 
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The author found that science attitudes neither accounted for any substantial variation in 
total score in booklet four; nor differentiated between the sexes. She believes rather than 
cultural background, societal factors and psychological factors are responsible for the high 
female response to the "I don’t know" answer. 

The results explained the gender differences in choosing of the "I don’t know" item by 
students as: 

1. differential prior instructions, 

2. differential response to uncertainty and the use of "I don’t know" response, 

3. differential response to "figurally" presented items, 

4. different attitudes toward science. 

Linn stresses that females were more likely to use "I don’t know" for such items with 
physical science or masculine themes such as football. The author feels that to ameliorate 
this situation there is need for more effective science instruction and more gender neutral 
assessment items. 

A study conducted by Erb and Smith (1984) measured the validity of adolescent males 
and females’ attitude toward women in science. The researchers used Women in Science 
Scale (WISS). 

WISS is a 27 item Likert-type instrument. The 6 point WISS scale ranges from "strongly 
agree" to "strongly disagree" with no neutral point. The importance of three dimensions 
of attitude toward women in science were used in the WISS: characteristics needed for 
science career pursuit; compatability of spouse, parent, and career roles; and equality of 
opportunity to pursue a career. The WISS was developed to evaluate COMETS (Career 
Oriented Modules to Explore Topics in Science). COMETS is a set of curriculum 
materials for grades 5 through 9. It is designed to enable science teachers to confront 
students with role models of women in science careers. It is a technique which had been 
shown to be a successful avenue for encouraging more girls to consider pursuit of science 
careers. The validity of WISS has been established by showing that it clearly 
distinguishes between scores of early adolescent males and females regardless of age or 
their construct of interest. 

WISS can be useful to evaluate a science career education program for early adolescents 
and to evaluate similar programs intended to effect attitudes toward women in science 
among both males and females in middle school/junior high. It can also be used to 
compare the attitudes toward women in science held by different groups of early 
adolescents. The subjects were differentiated by sex, race, socioeconomic status, grade 
levels, and locale (urban vs. rural). 

In Ontario (Connelly, 1987) it was found that in evaluating science achievement, males 
were better than females; but not by so much as evaluators thought they might be. The 
instrument used a three point scale: "agree", "uncertain", and "disagree". Connelly 
argues that our own culture limits our ability to sort out the "genetic", "environmental" 
and "educational" differences. The ideas of achievement are restricted by the kinds of 
knowledge and the "ways of knowing" which are socially valued and tested. The 
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differences in boys’ and girls’ achievement may simply reflect the differences in the way 
society has valued particular ways of knowing over others. 

5.3. Culture 

\ 

In evaluating STS courses or attitudes toward science, cultural differences must be taken 
into account. A number of researchers have concluded that evaluation instruments are 
culturally restricted (Khalili, 1987; Hofstein, Scherz and Yager, 1986; Zoller, 1987). 

Zoller attempted a comparison of B. C. students’ STS response profile, with that of 
American students. He found the comparison nearly impossible due to the different 
design, target populations, methodology, and purpose of the two studies. However, in 
probing into this issue through quasi-quantitative meta-analysis methodology, Zoller 
points out that the base line STS response profile of the two populations is significantly 
different. 

According to Zoller this is to be expected, since "STS issues" are contextually bound; that 
is, perception by the public is contingent on the particular local socio- cultural norms and 
realism of economic/political constraints. 

In designing STS science courses, the cultural background of the students must be taken 
into consideration. According to Smith (1986), native students in Manitoba have different 
needs that non-native students. Smith provides an analysis of work being done with 
native children in Manitoba to help them learn about the connections between myths and 
beliefs of their own culture and the science taught in school. Smith explores the 
theoretical perspective and practical examples of how this integration can occur. 

For example, Smith points out that native students have well developed observational 
skills. However, due to language and past experiences, they might state their 
observations differently than expected in the classroom. Furthermore, Smith stresses, 
curriculum and test development must take into consideration the cultural differences for 
native students. 

As mentioned above, students’ cultural background is a factor in curriculum development 
and evaluation. An interesting study conducted by Quinn and Kessler (1986) examines 
the scientific abilities of unilingual and bilingual adolescents, in south-western United 
States. The subjects were 6th grade students in two intact classrooms, one unilingual 
English speaking and one Spanish-English bilingual. Twelve science inquiry film sessions 
and six discussion sessions were used. All lessons were taught by the same teacher in 
English. Each film session based on a 3-minute film loop depicting a single physical 
science problem, ended with the students writing as many hypotheses as possible in a 
rigorously controlled 12-minute period. 

Q uinn and Kessler compared the ability of unilinguals and bilinguals to formulate 
scientific hypotheses; to write syntactically more complex language; and to generate 
multiple metaphors. 

The researchers believe that the ability to use science processes is a manifestation of 
cognition. They claim that bilingualism enhances that ability in at least one of its 
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aspects. While an aspect of cognitive ability is language development, the metaphorizing 
process is more apparent for bilinguals engaged in the process of formulating solutions to 
scientific problems than for unilinguals. 

The results of the study demonstrated that bilingualism appears to have a positive effect 
on the cognitive ability of students to formulate science hypotheses and to use language of 
increasing complexity. Bilinguals are also more capable of entering into the creative 
processes of divergent and convergent thinking. The generation of multiple solutions to 
science problems and hypotheses formulation are expressions of divergent thinking and a 
form of creativity. 

Quinn and Kessler feel that the enrichment that appears to accompany bilingualism is 
intimately related to the cultural universe or experimental domains of the students. The 
cultural background of the Mexican-American children in this study appears to effect the 
way they formulated scientific hypotheses in the creative, metaphoric use of language. 

The authors believe that a science education program which would require more precise, 
analytical language prematurely, would fail for these children to take into account 
cultural diversity and its contributions to creativity in language cognitive functioning. 

It must be mentioned that the above benefits of bilingualism would only occur if the 
subject is completely fluent in both languages. Otherwise, the subject would not be able 
to comprehend the complexity of the science topic. This study could be relevant to 
Canadian researchers exploring the multicultural facet of science education. In addition 
it must be noted that certain instruments of evaluation are culturally relevant to the 
subjects. Cultural differences between students and science teachers can also influence 
learning of the subject. 

5.4. Social Influence/pressures 

In studying attitudes, researchers have found that societal factors affect students’ 
behaviour in science classes. A study of decision-making theories as tools for interpreting 
student behaviour during a scientific inquiry was conducted by Glen Aikenhead (1989). 

In this experiment students attempt to resolve conceptual conflicts concerning a 
pendulum’s period by working towards a consensus. Aikenhead studied factors that affect 
conceptual changes in classroom decision-making. In particular he analyzed strategies 
students invented to maintain their alternative conceptions of motion related to the 
pendulum in the face of conflicting evidence. 



Aikenhead questions the degree that decision-making models accurately portray what 
students do when engaged in classroom decision-making. The results of his study show 
that conflict exists between sticking to one’s biases and believing the science instruction. 
According to Aikenhead, if a sociological theory could be refined in the simpler context of 
students making scientific decisions, then the theory may be applied to the context of 
students making the socio-scientific decisions found in STS instructions. In the STS 
context, students are expected to deal with both nonsocial cognition and social cognition. 



Therefore, Aikenhead believes that decision-making models from sociology and psychology 
are of little use in predicting the more complex decisions associated with socio-scientific 
Aikenhead stresses that scientists do not necessarily change their conceptions 



issues. 
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solely on the basis of new evidence and its explanation. Rather scientists often 
demonstrate a commitment to their conceptions. The same can be expected of students. 
Thus, decision can be swayed by: authority of data, authority of explanation, making 
sense out of the data, authority of teacher, the class social environment, psychological 
state of decision maker. Since decision-making is a part of the STS curriculum, this study 
is relevant in designing STS programs. 

The role of the media in students’ attitude toward science has been studied by Schibeci 
(1986). Schibeci is concerned with the portrayal of science and scientists in popular 
culture. Schibeci points out that literature and the media represent one of the many 
informal avenues for science education, along with museums, fairs, and exhibits. Of 
these, television appears to be potentially the most powerful of these informal avenues 
because children watch so much of it. Thus in assessing students attitude toward science, 
it should be remembered that outside influences (such as TV) are a factor in attitudes. 

5.5. Meta-analysis as a Research Tool 

Meta-analysis uses a quantitative approach. It is described by its originator as "the 
statistical analysis of the summary findings of many empirical studies" (Glass 1982, 
p. 200). This approach to the integration of the research literature has been rejected by 
some (Haig, 1988). Others such as Strube and Hartmann (1982), indicate the potential 
usefulness of the approach, provided its limitations are recognised. 

According to Schibeci (1984, p. 29), one line of investigation which could be pursued by 
those interested in meta-analysis would be a reliability trial. Two people (or two groups 
of people) would conduct, quite independently, the meta-analysis of the same broad area 
of concern. The results could then be compared. Schibeci notes that another, perhaps 
more important line of investigation, would be to examine the validity of the method- to 
what extent are the assumptions which underlie the method valid? Another note of 
caution which needs to he sounded is the ease with which the results of meta-analysis can 
be misinterpreted. 

6. CONCLUSION 

The incorporation of STS in science education at the secondary level is growing rapidly. 

As new programs come on stream, evaluators will continue to address the concerns and 
interests reviewed in this report. While this review cannot pretend to be comprehensive, 
since the field is so varied and changing, a number of summary conclusions can be drawn 
that are relevant to the Ontario context. 

6.1. Diverseness of STS 

The nature and role of STS in science education means different things to different 
people. Some science educators and curriculum developers see STS as a motivator, and 
secondary to science content. Others view STS as providing critical thinking and social 
skills. There is often tension between science teachers holding these different views. 
Science teachers that see STS as a motivator rather than the focus of a science course 
believe that science should be taught as method of knowing. On the other hand, there are 
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teachers that see science as a social institution and they think it should be taught as 
such. This conflict has to be kept in mind while developing science curricula as well as 
evaluation instruments. It should certainly be borne in mind when interpreting results 
from evaluation or monitoring instruments. 

There also exist opposing views regarding the relationship between science and 
technology. For most STS researchers, technology is seen as an independent entity from 
science. However, many educators (including the Ontario Ministry of Education) see 
technology simply as applied science. Thus, evaluators and researchers must be wary of 
the different philosophies that are brought into teaching the curriculum when interpreting 
results of evaluations. This is specially relevant, since students receive information and 
form opinions from material obtained outside the classroom. 



6.2. Culture 

Cultural diversity is an important factor in science education and STS evaluation. For 
example an instrument developed in Australia or even the United States might not be 
relevant to a Canadian setting. Instruments must be sensitive to the ethnic diversity in 
Canadian schools. 



6.3. Gender 

For an STS program to be implemented effectively, it must be gender neutral. The same 
is true for instruments used for STS evaluation. However, the role of women in science is 
an important part of STS education and should not be ignored. Furthermore, the 
presentation of role models for female science students may encourage women to enter 
careers in science. 



6.4. Local Relevance 



STS implementation and evaluation must be locally relevant. Studies have shown that 
students relate better to STS issues when presented with specific examples dose to home. 
For the Ontario context, this means that STS evaluation must reflect the various regions 
of the province. In addition, there should be national relevance for students as ritizens of 
Canada. 

These considerations do not predude an international context being present in STS 
teaching and evaluation. As the world becomes smaller because of the technological 
revolution, the role of srience and technology must be looked upon in its global context. 
For example, environmental issues that effect us regionally are also important in the 
international context. This dichotomy must be reflected in STS programs and evaluation. 

In conclusion, it is appropriate to come back to the diverse nature of the STS field. The 
perception of science and technology as integral parts of the sodal fabric is relatively new 
in srience education, especially at the secondary level. Alongside teachers and curriculum 
developers with a fairly sophisticated understanding of the role of srience and technology 
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in society, are those who hold quite positivistic views about the primacy of scientific 
thinking and its importance for technological and economic progress. Add to this mix the 
variations of culture, regional interests, training, motivation and pedagogical philosophy 
that characterize Ontario’s teaching professionals and student body. And also consider 
the fact that students pick up a great deal of their information and opinions • perhaps 
indeed the majority • from outside sources such as television, newspapers and books. One 
can only conclude, therefore, that the STS component of the Ontario science curriculum 
will be taught by teachers and learned by students in a multiplicity of ways. 

Hence one must appreciate the fact that no one instrument can possibly measure all the 
subtleties of students’ understanding of STS issues. The best attempts will undoubtedly 
be made to develop monitoring and evaluation instruments that are gender neutral, 
sensitive to cultural diversity and inclusive of various views regarding the nature and 
importance of science and technology. However, interpreters of the data must bear in 
mind the variations that exist in the STS field itself, and the teaching and learnin g 
environments in which it is taught and learned. 

The best use of data obtained from an instrument of a pool of questions that are used to 
measure students’ views on STS, is to establish a baseline from which to monitor 
progressive change. 

7. FOOTNOTES 

1. There are also summations of these reports in McConnell’s article: national 

science foundation status of science education studies, national science foundation 

and department of education joint report on science and engineering education 
(1980), status study of graduate science education in the united states 1960-1980, 
the advisory committee for science education of the national science foundation’s 
report to the national science board (1980). 
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APPENDIX "A" 



FINAL TEST INSTRUMENT USED 
IN PROVINCE-WIDE TEST 
MARCH-APRIL 1990 



ONTARIO SCIENCE AND TECHNOLOGY SURVEY #B483 Market Facts of Canada Limited 

conductecTby 77 Bloor West, Toronto MSS 3A4 

MARKET FACTS OF CANADA UMITED (416)964-6262 



Thank you very much for taking part in this survey amontj students in Ontario. The main purpose of 
this study is to find out how the present generation of high school students like yourself feel about 
matters related to science and technology. 

In any effort to understand what science and technology mean to Canadians, the views of students 
are particularly important. You in fact represent Canada's future and your feelings today will 
influence attitudes towards various issues (like science and technology) in the years to come. 

Please give us your opinion on each of the subjects in the questionnaire. There is no identification 
required. You are therefore assured of complete anonymity. 

Please put your completed Questionnaire in the envelope provided and seal it before handing it in. 
All envelopes will be opened and the questionnaires tabulated by Market Facts of Canada Limited to 
ensure the anonymity of those taking part in the survey. 

INSTRUCTIONS FOR COMPLETING QUESTIONNAIRE 

This questionnaire is made up of two parts. As you can see the first part is divided into a number of 
individual boxes, each containing 3 or 4 sentences. 

Each box deals with one aspect of science and technology and the sentences in the box express 
different ideas or opinions on this. FOR EACH BOX, please read all the sentences and then cirde the 
number next to the one that comes dosest to your opinion. 

In the lower half of each box, space has been provided for you to write in your feelings on the 
subject. 

Before you go on to answer the questionnaire please complete the section below. This information 
is needed for tabulation only. 



Cd. 1 

AGE in years I I 1 
6- 7- 

SEX: Male Qi ~ 8 

Female ... Q2 



Name of school: 

Type of school: Public C]1 

City/town in which located: 



GRADE/LEVEL: Grade 10 Basic Sdence ; Q 1 

Grade 10 General Sdence . .. Q2 
Grade 10 Advanced Sdence . □ 3 

Grade 12 Basic Sdence Q4 

Grade 12 General Sdence ... QS 
Grade 12 Advanced Sdence . Q 6 

10 bl. 

Parochial Q2 



12 - 

13- 



14-1 

15*1 



16-19 bl. 



ERIC 
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